Strategy for communicating benefit-risk decisions: a comparison of regulatory agencies' publicly available documents

The assessment report formats of four major regulatory reference agencies, US Food and Drug Administration, European Medicines Agency, Health Canada, and Australia's Therapeutic Goods Administration were compared to a benefit-risk (BR) documentation template developed by the Centre for Innovation in Regulatory Science and a four-member Consortium on Benefit-Risk Assessment. A case study was also conducted using a US FDA Medical Review, the European Public Assessment Report and Australia's Public Assessment Report for the same product. Compared with the BR Template, existing regulatory report formats are inadequate regarding the listing of benefits and risks, the assigning of relative importance and values, visualization and the utilization of a detailed, systematic, standardized structure. The BR Template is based on the principles of BR assessment common to major regulatory agencies. Given that there are minimal differences among the existing regulatory report formats, it is timely to consider the feasibility of a universal template.


INTRODUCTION
The evolution of the requirements for assessing the benefits and risks of medicinal products has resulted in changes in the regulatory review process. Beyond the separate assessment of benefits and risks, the emphasis is now on the balance between the two, having to justify the potential harms in view of the efficacy claims (Breckenridge, 2010). This balance can be expressed in a transparent manner using a structured framework that aids in the communication of the differences in opinions between regulators and the pharmaceutical industry. In a changing society where the demand is for transparency of such decision-making processes, there is now a major challenge to adequately communicate the relevant information to stakeholders.
As expectations of stakeholders change with the rapid advancement of science, regulatory agencies need to adapt to meet these changing requirements. In the European Medicines Agency (EMA) Roadmap to 2015, one of the strategic areas identified was the reinforcement of the benefit-risk balance assessment model through a set of priority activities (European Medicines Agency, 2011a). These included looking at appropriate quantitative tools, improving the quality and consistency of the outcomes, and reviewing the European Public Assessment Reports (EPARs) to improve communication of benefit-risk decisions to stakeholders and to increase the involvement of patients, academia and healthcare professionals. Similarly, since 2009 the US Food and Drug Administration (FDA) have begun initiatives to explore systematic approaches to assess and communicate benefits and risks. Their initiatives have included the development of a framework to characterize and provide a structure for the benefit-risk assessment already existing in their decision-making processes, as well as communicating the reasoning behind the decision to all stakeholders (US Food and Drug Administration, 2012), which has led to the current FDA five-step benefit-risk framework. These steps are related to the five key areas to be discussed in the assessment of a medicine, namely the analysis of the condition, the medical need for the product, clinical benefits, risks and risk management. Mullin, commented that this structured framework has the potential to improve the predictability and consistency of decision-making as it is capable of clearly outlining both the available evidence and the uncertainties (Centre for Innovation in Regulatory Science, 2011). This would enable the US FDA to articulate the consideration and clinical judgment taken for the benefit-risk decision and hence improve the transparency of the decision-making process (US Food and drug Administration, 2013). The US FDA has also published its own user's guide on communicating benefits and risks, which provides the expectations and standards for communication (US Food and drug Administration, 2011). Similarly, Australia's Therapeutic Goods Administration (TGA) will be focusing on increasing transparency through their new initiatives and engaging stakeholders with a new framework for communicating the benefits vs. risks approach in their regulation of medicines (Therapeutic Goods Administration, 2013). This increase in transparency is to be achieved through information that is easily understood by patients and consumers and shared with healthcare professionals. TGA aims to provide accessible, clear and consistent relevant information through various multimedia platforms.
The results from a recent study (Leong et al., 2013) showed that both regulatory agencies and pharmaceutical companies believe that a benefit-risk framework would enhance the quality (transparency and consistency) of communication and should provide documentation for a structured discussion, acting as a tool for communication among peers within an organization and between the organization and other stakeholders (Centre for Innovation in Regulatory Science, 2012). This eight-step universal benefit-risk framework encompasses the principles of existing frameworks by other major regulatory agencies such as the US FDA (US Food and drug Administration, 2013) and EMA (European Medicines Agency, 2010) ( Table 1).
In collaboration with CIRS, TGA, the Singapore's Health Sciences Authority (HSA), Health Canada and SwissMedic established a four-member Consortium on Benefit-Risk Assessment (COBRA), which aimed to develop a systematic qualitative approach for the benefit-risk assessment of medicines in order to facilitate joint and shared reviews. A benefit-risk (BR) documentation template, or BR Template, was developed based on the EMA reflection paper (European Medicines Agency, 2008) and reviewed by the Consortium through both retrospective and prospective studies employing its use, with plans for making the template more reflective of actual practice. This BR Template was designed to enhance effective documentation and communication of decisions and was used as the basis of a comparison in this study. The summary portion of this template was extracted to produce a "stand-alone" BR Summary Template, with the objective that this simplified template would suffice to meet the needs of jurisdictions with emerging pharmaceutical markets. This study aims to review the publicly available assessment reports to see if they adequately fulfill the functions found in the BR Template and BR Summary Template.
The objectives of this study were to: • Compare the format of publicly available assessment reports from US FDA, EMA, Health Canada and TGA with the BR Template and BR Summary Template • Evaluate whether these four regulatory agencies have an effective approach to communicating benefit-risk decisions to all stakeholders • Examine the utility of the BR Summary Template for communicating benefit-risk decisions by the US FDA, EMA and TGA using a case study approach

DEVELOPMENT OF THE BR TEMPLATE AND BR SUMMARY TEMPLATE
A benefit-risk documentation template, or BR Template, was developed based on the EMA reflection paper (guidance document, 2008). This document contained those elements considered essential to the assessment of benefits and risks of medicines. These elements were then transformed into a template that allowed systematic documentation and editing. This initial developmental template was then reviewed against the universal framework so that it could support its underlying principles. The Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Universal benefit-risk framework template comprises six sections namely: Section 1, Background (decision context); Section 2, Overall Summaries (quality, nonclinical, human pharmacology, clinical); Section 3, Identified Benefits and Risks (list of all benefits, list of all risks); Section 4, Benefit-Risk Study Findings (benefits observed in pivotal and nonpivotal studies, overall summary of risks, adverse effects/events, and uncertainties); Section 5, Benefit-Risk Summary Table  and Expert Judgment (benefits-weighting/relative importance and valuing, risks-weighting/relative importance and valuing); and Section 6, Benefit-Risk Conclusions. This initial template was assessed by the Consortium who evaluated its use in a feasibility study, which led to minor amendments of the template incorporating all the feedback from the Consortium. This BR Template was then applied in a retrospective study followed by a prospective study for the review of a number of products, which resulted in the final version of the Benefit Risk Template. It was considered beneficial by the Consortium that the addition of a Summary section to the BR Template would enhance its internal and external value for communication purposes to stakeholders. Therefore, a Summary section was developed by the expert reviewers who identified the key features from the BR Template (benefit-risk conclusions, decision context, identified benefits and risks, benefit-risk: weighting/relative importance and valuing and benefit-risk management). It was later recognized that the BR Template was going to be more relevant to the review process employed by the established agencies, but the Summary Template, as a shorter version, would be of more benefit to the maturing agencies. The Summary Template has subsequently been validated by several agencies that have carried out retrospective reviews of a number of products.

COMPARISON OF THE PUBLICLY AVAILABLE DOCUMENTS FROM FOUR REFERENCE AGENCIES
In order to establish the utility of the BR Template, reference agencies were selected based on the criteria of a positive history, global recognition and the public availability of assessment reports. For the purpose of comparison in this study, the following report formats of the four reference agencies were selected: • US FDA-Medical Review and the Benefit-Risk Assessment • EMA-European Public Assessment Report and the Executive Summary • Health Canada-Summary Basis of Decision • TGA-Australian Public Assessment Report Report formats were retrieved online for each agency and where appropriate, a recent publicly available assessment report would be used to review the contents of the format or support the understanding of the format. A comparison of the report formats from the four reference agencies was conducted by reviewing the section headings of the report against those of the BR Template and BR Summary Template. Where there was a summary in the reference agency format, this would be directly compared with the BR Summary Template and the findings are tabulated and presented.
Furthermore, to illustrate the use of the BR Summary Template, a case study was conducted using a recent US FDA Medical Review (European Medicines Agency, 2013a), EPAR (Therapeutic Goods Administration, 2014) and AusPAR (European Medicines Agency, 2011b) for the same product. Zivaflibercept, indicated for the treatment of metastatic colorectal cancer, was chosen as it was approved around the same time by the agencies (03 August 3, 2012 for US FDA, February 1, 2013 for EMA and April 2, 2014 for TGA). Importantly, the US FDA Medical Review was written according to the new five-step benefit-risk framework that features the Benefit-Risk Assessment. These two respective summaries were transferred into the BR Summary Template and the omissions reviewed. This entire study was designed to be exploratory and the outcomes were interpreted to provide qualitative inferences relating to the aims. No statistical analyses were planned or conducted.

RESULTS
The outcomes are presented in three parts: • Part I-Formats of the four reference agencies' publicly available reports • Part II-Comparison of the four reference agencies' report formats with the BR Template and BR Summary Template • Part III-Case study of US FDA, EMA and TGA summary reports on ziv-aflibercept (Zaltrap®; Regeneron Pharmaceuticals, Inc./Sanofi-Aventis U.S. LLC, Bridgewater, NJ, USA)

US FDA medical review
The US FDA Medical Review consists of nine sections (Table 2), with the opening section presenting the recommendations and benefit-risk assessment (based on the five-step benefit-risk framework). The remaining sections present the details of the assessment supporting the recommendations. It is known that the publicly available reports from US FDA are a redacted subset of the complete evaluation data. The original dataset would have included discussions of queries and responses by the sponsor with the US FDA. The US FDA five-step benefit-risk framework appears to closely reflect the format of its benefit-risk Assessment (Table 3), and therefore was compared with the BR Summary Template. Although the five-step benefit-risk framework may appear less comprehensive than other existing frameworks, the US FDA is currently reviewing a list of questions that should be included under each of these steps, in an approach similar to the EMA guidance for the assessment of benefits and risks.

EMA EPAR
The EPAR consists of an Executive Summary and four sections ( Table 4). The publicly available EPAR is extracted from the complete assessment report which would have included responses and justifications to EMA for queries raised.

Health Canada summary basis of decision
The Health Canada's Summary Basis of Decision (SBD) consists of eight sections ( Table 5). This is a publicly available document that presents the relevant information to support the decision www.frontiersin.org December 2014 | Volume 5 | Article 269 | 3

TGA Australian public assessment report
The TGA Australian Public Assessment Report (AusPAR) consists of six sections (Therapeutic Goods Administration, 2012) ( Table 6), the format being close to the EPAR but without the Executive Summary. As with the formats of the other three agencies, agency-specific information is that related to individual regulatory and submission information. It is known that the AusPAR contains information extracted from the complete, original assessment reports.

PART II-COMPARISON OF THE FOUR REFERENCE AGENCIES' REPORT FORMATS WITH THE BR TEMPLATE AND BR SUMMARY TEMPLATE
The formats of the reference agencies' reports are generally similar when compared with the BR Template (Walker et al., 2014). They were all found to lack the features that list: (1) the identified benefits and risks; (2) application of values and weights (relative importance); and (3) visualization of the assessment outcomes (Supplementary material). In addition, while it is acknowledged that relevant discussions and considerations contributing to the final benefit-risk decision maybe reported in the existing reference agencies' reports, the BR Template allowed for this through a structure of guided questions.
In comparison with the BR Summary Template, it was found that only two reference agencies have defined summaries within the report. The US FDA Medical Review has the Recommendations/Benefit-Risk Assessment, which is a

US FDA medical review
The approach of the US FDA Medical Review and the BR Template was found to be similar, with the focus on the contribution of clinical efficacy and safety to the overall benefitrisk balance and a significant contribution of quality, non-clinical and pharmacology concerns succinctly discussed. Overall, it was observed that the US FDA Medical Review was designed to present details of the evaluation processes including those of the studies and considerations, while the BR Template presents only the information that will directly contribute to the decision on the benefit-risk balance. This can be seen in the detailed structure of the US FDA Medical Review, compared with the more concise BR Template. In terms of utility, the BR Template and BR Summary Template appear to share the US FDA Medical Review's capability to present critical information regarding the benefit-risk decision. The more explicit display using the BR Template's method of listing identified benefits and risks, use of weighting, valuing and visualization may facilitate an improved outcome through a more structured format on the discussion for benefit-risk balance and therefore enhance communication to stakeholders for whom this information is important. Comparison with the BR Summary Template behaved exactly the same as the BR Template exhibiting the same attributes.

EMA EPAR
In assessing the benefit-risk balance, the BR Template provided a more structured format through the use of guiding questions, while the EPAR was a general descriptive write-up. The deficiencies seen in the EPAR were found to be similar to those observed for the US FDA Medical Review.
The Executive Summary of the EPAR was compared with the BR Summary Template, which presents structured, concise information leading to the benefit-risk decision, exceeding the utility of the Executive Summary in the EPAR. Therefore, the BR Summary Template may communicate the outcomes in a more transparent manner than the Executive summary in the EPAR.

Health Canada SBD
In the assessment of efficacy and safety, it appears that the SBD does not provide a detailed structure in presenting information compared with the BR Template. Overall, both the SBD and BR Template are comparable for the documentation of clinical efficacy, safety and benefit-risk assessment. The SBD, however, did not provide a list of benefits and risks that were considered and subsequently excluded or included in the benefit-risk assessment. Again, weighting (relative importance), valuing and visualization were not included, which would improve transparency as well as communicating the basis of decision. These deficiencies were similarly observed when the SBD was compared with the BR Summary Template.

TGA AusPAR
As observed in the other three publicly available documents, the AusPAR was deficient in that it did not provide a list of benefits and risks that were considered, nor did it include their relative importance, their values, or any visualizations. As observed with It was observed that there was no defined summary for the AusPAR, the section on overall conclusion and the benefitrisk assessment appears to function similarly to the US FDA Recommendations/Benefit-Risk assessment and EPAR's executive summary. The AusPAR appears to represent most of the functional sections of the BR Summary Template, although it still has the deficiencies detailed above.

PART III-CASE STUDY OF THE US FDA, EMA AND TGA SUMMARY REPORTS ON ZIV-AFLIBERCEPT
Ziv-aflibercept was approved by both US FDA and EMA and the publicly available Medical Review (US Food and Drug Administration. Medical Review, 2012) and EPAR (European Medicines Agency, 2013a) were retrieved from the internet. Both of these documents provided similar information ( Table 7), but presented in a different manner. The Executive Summary was written in a continuous descriptive prose but the US FDA Benefit-Risk Assessment was presented under six headings. Overall, the BR Summary Template is more structured in presenting the information for the benefit-risk decision.

US FDA benefit-risk assessment
The BR Summary Template was completed with the information (Annex C in Supplementary Material) from the Benefit-Risk Assessment for ziv-aflibercept. The decision context of the BR Summary Template could be sufficiently completed with information from this report. Conclusions on quality, non-clinical, and human pharmacology were absent from the Benefit-Risk Assessment.
The benefits were included for the benefit-risk assessment but no reasons were provided for their inclusion. There was no information provided on those benefits that were reviewed but subsequently excluded. Safety parameters included could be inferred by the reasons provided in the Benefit-Risk Assessment but the risks reviewed and subsequently excluded were not documented. Weighting (relative importance) and valuing were not documented and there were no specific comments on the uncertainties relating to the listed benefits and risks. The section of the BR Summary Template (Section 7 of Annex B in Supplementary Material) could be completed from this report although not presented in a structured manner. Overall, application of the BR Summary Template to the US FDA Benefit-Risk Assessment provided a more structured and guided discussion of the decisions leading to the eventual benefit-risk balance.

EPAR executive summary
The BR Summary Template was completed with the information (Annex C in Supplementary Material) from the EPAR Executive Summary. The Executive Summary has no structure and is presented in a single section. The quality, non-clinical, human pharmacology conclusions of the BR Summary Template could not be completed as they were absent from the Executive Summary and this is a similar situation to that observed with the US FDA Benefit-Risk Assessment. As there was no specific safety summary, the clinical conclusion of the BR Summary Template remained incomplete. The benefits included for benefit-risk assessment are presented in the Executive Summary of the EPAR but there were no reasons for their inclusion. In addition, there were no indications for those benefits excluded or reasons for their exclusion. A similar pattern was observed for the safety parameters. Sufficient information was provided in the Executive Summary to complete the BR Summary Template clinical study information and tables, but no weights, values or comments on uncertainties were available. The above observations are expected as relative importance is carried out implicitly but not explicitly in many agencies. However, the safety exposure information was written entirely as a paragraph and could not be uploaded as an image into the BR Summary Template. The structured discussion of the BR Summary Template was not adequately completed using the Executive Summary.

TGA overall conclusion and risk/benefit assessment
The BR Summary Template was completed with information from the Overall Conclusion (Annex C in Supplementary Material). This last section of the main AusPAR (Therapeutic Goods Administration, 2014) is organized into seven headings, namely background, quality, non-clinical, clinical, risk management plan, benefit-risk analysis and outcome. Compared with BR Summary Template, it appears that it may have an advantage over this summary by expressing the required information in a succinct way.

DISCUSSION
The comparisons conducted in this study have shown that the formats of the publicly available assessment reports from the four reference agencies are similar and generally allow the information generated through the course of the evaluation to be described. The differences between these reports are largely related to their format arrangement and headings provided for each section. While there is no universal template for an assessment report, there does not appear to be major differences as to how such information should be presented. Thus, given the commonalities with their presentation, only minor changes are required to their current formats in order to achieve a potential universal standard structure.
The publicly available assessment reports are the means for documenting the relevant information made available to stakeholders and to communicate the basis and justification for BR decisions. The US FDA has made a recent attempt through PDUFA V, to provide a Benefit-Risk Assessment, based on their five-step framework, which details their considerations contributing to the regulatory decision and features an additional succinct benefit-risk assessment summary. EMA has commissioned an external expert to improve its communication of benefits and risks (European Medicines Agency, 2011b) and the EPAR features an Executive summary to provide concise information. Health Canada has completed its two phases of the initiative to improve documentation and communication to the public, with an emphasis on the discussion of the benefit-risk balance and the basis of the decision (Health Canada, 2012b). Likewise, TGA has commenced a project targeted at improving communication of information to patients and physicians. Therefore, it is concluded that these agencies recognize the need to effectively communicate the basis of their decisions through a concise documentation tool.
The primary concern of most patients would be to know if a product is effective and safe, while the physicians would want to know the details to make a better informed decision when choosing an optimal treatment for their patients. Therefore, clarity in the presentation of such information is of paramount importance. For pharmaceutical companies, a documented transparent decision-making process would enable them to understand the basis of the regulatory decision, the rationale for the inclusion or exclusion of benefits and risks, as well as the final benefitrisk balance. This would, therefore, provide a suitable platform to discuss any discrepancies in interpretation or differences in opinions. In evaluating the product for pricing and reimbursement, HTA agencies would also want to understand the rationale for the approval of a product. A failure to achieve this understanding might affect a product's accessibility for patients and influence the healthcare delivery in terms of cost and clinical management. Assessment reports of major regulatory agencies are often accessed by smaller agencies in the emerging markets to support their local decisions and thus these regulatory agencies should also be considered as key stakeholders for the publicly available assessment reports.
Certain jurisdictions may require publication of the assessment reports as a move to increase the transparency of the decision-making processes while others may require varying amounts of information to be made public. As previously discussed, it is also not known if the current practices of providing the publicly available assessment reports actually achieve the transparency required or desired by the stakeholders as there are no studies describing this type of feedback from pharmaceutical companies, physicians, patients or regulatory agencies. In fact, the vast amount of unstructured information provided may possibly hamper understanding and thus communication. The use of summaries like the Executive Summary of the EPAR, the Benefit-Risk Assessment of US FDA and Overall Conclusion of the AusPAR aims to further improve communication. However, as seen in this case study of ziv-aflibercept, a more structured and guided discussion, such as that provided by the BR Summary Template, may further help to improve both transparency and communication and prevent the omission of information assessed by the reviewer and deemed important to stakeholders. The comparison of the summaries showed that there are elements missing which could facilitate effective communication. As such, the elements from the BR Summary Template found missing in the summaries of the reference agencies may serve as a starting platform to enhance the effectiveness in communicating benefit-risk decisions. Furthermore, the provision of a list of identified benefits and risks and visualizations would aim to facilitate communication by reducing the amount of text needed to convey these messages.
As a result of this study, future attempts to improve the quality of communication should include the following: • A listing of benefits and risks, with justification for their roles in assessing the benefit-risk balance and the reasons for their inclusion or exclusion • Valuing the identified benefits and risks • Weighting (relative importance) of the identified benefits and risks • Providing visualizations of the outcomes • Utilizing guided discussions and structured questions such as deliberations on uncertainties, consistency of outcomes across studies and additional risks compared with a standard of care to illustrate key discussion points leading to benefit-risk decisions Given that there are minimal differences among the existing formats of the reference agencies, it is timely to consider the feasibility of a universal template. The BR Template and BR Summary Template were based on the EMA reflection paper (European Medicines Agency, 2008) for the assessment of benefits and risks and to allow documentation of these considerations in support of the decision. Unlike the existing formats, the guided discussion, structure, listings of identified benefits and risks, application of values and weights and visualization of the BR Template serve to improve effective communication. Familiarity with a standard template and its presentation format will enhance the stakeholders' experience in seeking to understand the key messages. A universal framework for the assessment of benefits and risks will be required to bring focus to the agencies, which would then facilitate the implementation of a standard, universal documentation tool. An eight-step universal benefit-risk framework has been developed that incorporates the existing frameworks of major regulatory agencies and those used by pharmaceutical companies (US Food and Drug Administration. Medical Review, 2012). Given that the BR Template and BR Summary Template was developed using the principles from this universal framework, there is now the opportunity to explore their universal use. However, as the basis for publicly available assessment reports, it would be prudent to seek more confirmative opinions from stakeholders on the feasibility and utility of such an initiative through further studies.
In the course of this research, some areas for improvement were identified for the BR Template and BR Summary Template. These included expanding the discussion on pharmacovigilance and RMP/REM, which would then align to the recent requirements for periodic benefit-risk evaluation report (PBRER) (European Medicines Agency, 2013b) and the emphasis on postmarket activities. As stakeholders are increasingly seeking the acknowledgment of their opinions, there should also be dedicated and defined areas for inputs from the various stakeholders, particularly patients. These improvements may enable the BR Template to accommodate requirements in the post-marketing setting as well as functioning as a tool for product life cycle management. If used as a universal template, it could trace and document the evolution of the benefit-risk balance of a product and provide meaningful comparisons using valid baselines. Ultimately, this may translate to an increase in consistency, transparency and the quality decision-making.