As for any drugs, the US Food and Drug Administration (FDA) is asked to consider the regulatory approval of cancer drugs based on “substantial evidence” of efficacy from adequate and well-controlled investigations.1 Historically, the efficacy of oncology drugs has been assessed based on objective tumor response rates, overall survival, time to tumor progression, disease-free survival, and progression-free survival.1,2 These end points, on which drug approval decisions are made, are still important today.1,2 Patient-reported outcomes (PROs) have long been standard medical practice; however, only after the Rand Health Insurance Experiment (conducted in 1974-1982)3,4 and the Medical Outcomes Study5 (conducted in 1986-1989), which together demonstrated that interpreting traditional clinical outcomes within the context of provider and patient reports could improve patient care, did the use of PROs in clinical research to demonstrate treatment efficacy become more common. In 1985, the FDA published a position paper encouraging drug manufacturers, particularly with respect to oncology drugs, to include patient-centric outcomes (symptoms and performance status) in their studies and “strongly urged” drug makers to discuss their study design and measurement strategies with the FDA before conducting their pivotal studies.6 Since then, several guidances (including those published by the FDA2,7 and the European Medicines Agency8,9) and methods,10,11 reviews,12,13 and original research papers14 have been published about the development and implementation of PRO questionnaires and the interpretation of PRO data.
Despite the growing attention to PROs in the peer-reviewed literature and by regulatory bodies, and the growing use of PRO measures in oncology trials,15 leveraging the patient perspective in drug approval decisions and reflecting the patient voice in drug approval documents has been challenging, as is evident by the lack of the patient perspective in US cancer drug labels.16 Acknowledging this disparity, Kitchen and colleagues provided considerations to help oncology drug developers maximize their chances in achieving PRO-based label claims, including that drug sponsors systematically plan for PRO data collection by providing a clear rationale and blueprint for the PRO measurement strategy.17
The objective of our article is to provide oncology researchers, and those involved with building oncology drug development programs, with tools to systematically plan for PRO data collection, particularly in relation to US drug labeling. We therefore review (1) regulatory and measurement terminology to provide a context for describing the use of PROs to test efficacy hypotheses in oncology clinical trials; (2) challenges drug sponsors face with a PRO-based measurement strategy; and (3) research tools and strategies that outcomes researchers can use to overcome those challenges.
Outcomes-Focused Nomenclature for Oncology Researchers
Much has been written regarding the use of PROs in oncology trials. The use of PROs is a positive advancement; however, clinical researchers are confronted by the inconsistent and/or erroneous use of measurement nomenclature in the published literature and within the healthcare industry.18 The lack of consistent language poses unique challenges for researchers in cancer trials because of the relative novelty of PROs in that space.
Given that this nomenclature applies to scientists representing clinical, commercial, statistical, medical, and regulatory perspectives, it is not surprising that this terminology may not be familiar to researchers. It is critical for drug developers to align their scientific, regulatory, and commercial goals with their measurement goals; key relevant terms are therefore discussed in this article.
In addition to legitimizing the use of PROs and other clinical outcome assessments as the basis for evaluating new medical treatments, the FDA’s “Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims” (hereafter referred to as the PRO Guidance) describes the type and quality of evidence that researchers need to substantiate the use of a given PRO questionnaire to support their claims in drug labeling.19
This PRO Guidance also includes a glossary from which many terms have been adopted and expanded in other US regulatory guidances, including the “Clinical Trial Endpoints for the Approval of Non-Small Cell Lung Cancer Drugs and Biologics”2 and the “Drug Development Tools Qualification Programs.”20 Similar glossaries have been listed elsewhere,21 and the International Society for Quality of Life Research built a Dictionary of Quality of Life and Health Outcomes Measurement,22 which was subject to public review and comment.
The FDA defines a drug label as “the official description of a drug product which includes the indication (what the drug is used for); who should take it; adverse events (side effects); instructions for use in pregnancy, children, and other populations; and safety information for the patient.”23 However, drug labeling conveys a more comprehensive meaning; it refers to “all labels and other written, printed or graphic matter (1) upon any article or any of its containers or wrappers, or (2) accompanying such article,” and typically encompasses any material containing drug information that is disseminated by or on behalf of the drug sponsor/manufacturer.24
Understanding the terms “label” and “labeling” in this context is important, because the PRO Guidance applies to how the FDA “reviews and evaluates existing, modified, or newly created patient-reported outcome instruments used to support claims in approved medical product labeling.”19 In other words, if a drug sponsor wishes to use a PRO for inclusion in a drug’s label or for labeling purposes, it is then subject to the nonbinding recommendations in the FDA’s PRO Guidance.
Whereas “label” defines the communication vehicle(s) for medical drugs, “claim” is defined as “a statement of treatment benefit. A claim can appear in any section of a medical product’s FDA-approved labeling or in advertising and promotional labeling of prescription drugs and devices.”19 The latter point of this definition supports the notion that the FDA statement that specifies the drug indication and the statements related to the drug’s treatment efficacy are the drug label claims (Table 1).
The terms “label” and “claim” imply a focus on regulatory science, whereas terms such as “concept,” “outcome,” “questionnaire,” “instrument,” and “end point” focus on measurement science or the science of self-reporting.25 Table 2 provides definitions for these and other key measurement terms.
The term “end point” is particularly troublesome, because it is often used interchangeably (albeit mistakenly) with terms such as “concept” (eg, my end point is pain), “outcome” (eg, my end point is pain reduction), and “questionnaire” (eg, my end point is the Brief Pain Inventory-Short Form [BPI-SF]26). However, the FDA defines the term “end point” as “the way an assessment will be used as a study result and statistically compared among treatment groups to assess the effect of treatment. Endpoints are often named by the assessment measured, but a complete statement of the endpoint should include a full description of what data are collected, and how they are analyzed to support a specific study objective.”27 In this way, an appropriately stated end point conveys significant information for outcomes researchers and may appear, for example, as a proportion of patients in the treatment groups versus comparator groups who have a ≥25% reduction in the BPI-SF Item 3 score from baseline to week 18 (Table 2).
The challenges facing clinical researchers in oncology are particularly difficult regarding the integration of measurement or PRO-focused language into their nomenclature. One reason for this is the historically simple concepts and end points that are required to support drug labeling. In oncology, the desired outcomes (eg, reduced tumor size) and, therefore, the sought-after claims for labeling (eg, our treatment reduces tumor size), are relatively straightforward. The measurement concepts (eg, tumor size), methods of assessment (eg, magnetic resonance imaging [MRI] or computed tomography [CT] scan), and end points (eg, proportion of patients with a ≥30% reduction in tumor size from baseline to week 16 as measured by MRI/CT scan) are also relatively straightforward.
Confusion exists regarding how to best articulate a PRO measurement strategy.12,28,29 Nevertheless, given the importance of PROs to characterize the patient experience for cancer drug approvals, it is critical for oncology researchers to become familiar with the PRO measurement nomenclature. A simple, clearly defined measurement strategy that is consistent with, and supportive of, the drug’s value is critical for drug developers and regulators,17 as well as for other stakeholders, including payers,15,30 providers,13,31,32 and patients.13,28
Articulating a PRO Measurement Strategy
The FDA has recommended the use of an end point model to articulate a clinical program’s measurement strategy in a way that conveys an end point’s relationship to a drug’s efficacy goals or sought-after labeling claims (Table 3).19 Although sponsors vary in terms of how they prefer to present their end point models, it is critical to specify what the sponsor wishes to say about a treatment (ie, the desired outcome or claim), what will be measured to test that claim (ie, the measurement concepts), how the concepts will be measured (eg, the PRO questionnaire), and how the measurements will produce study results that can be statistically evaluated to inform decisions regarding treatment effects (ie, the end point). A well-specified end point model should convey the relationships among all primary and secondary end points proposed in a trial, and is the blueprint for the measurement strategy.
Given that the end point model is the foundation on which regulatory decisions regarding the efficacy and safety of drugs are made, the significance of a thoughtfully constructed and accurately depicted end point model cannot be understated. Despite its importance, however, rarely is this information simply conveyed, and even the end point model examples provided in the PRO Guidance do not include actual end points.19 Other researchers have also noted that there are significant deficiencies in defining and reporting end points, even among the most often used end points in oncology.33-35 A poorly defined end point model (and thus a measurement strategy) may delay, or halt altogether, drug development and regulatory approval,36 and may impede the translation of knowledge into patient care.37
Table 3 provides a hypothetical end point model. As an extension of the examples provided in the PRO Guidance,19 this sample breast cancer end point model serves several purposes. First, it illuminates the link between a hypothesized drug claim, the concept of measurement necessary to evaluate that claim, the quantitative end point that will be used to generate the evidence that reviewers will need to make conclusions regarding the veracity of that claim (including the name of the assessment, what data are collected, and how the data are analyzed), and the type of assessment used for measurement. Second, by showing the hierarchy of primary and secondary efficacy end points, the drug’s “value story” can be easily explained.
Based on Table 3, for example, the efficacy story may be conveyed as follows: when administered to patients with breast cancer, treatment X reduces tumor size by X% and often leads to patients living longer than patients receiving comparator treatment Y. Moreover, patients who receive treatment X show greater reductions in their cancer-related symptoms of pain, fatigue, and impact on physical function compared with treatment Y. In addition, patients who receive treatment X have fewer treatment-related side effects compared with patients receiving treatment Y.
Overcoming PRO Measurement Challenges
Although a clear and accurate end point model is a step toward successful drug labeling in the United States, drug sponsors are rarely prepared to build such a model in early drug development. In oncology clinical trials, the focus on survival and tumor response is not surprising, given that cancer can be fatal. However, the increased attention to clinical outcomes and the FDA’s commitment to consider these outcomes as part of a new drug’s overall risk–benefit evaluation means that researchers must find ways to successfully integrate these outcomes into the clinical program if the ultimate stakeholder, the patient, is to benefit. Unlike survival and tumor size (which are evaluated through well-established clinical measures),1,2,28 outcomes such as symptoms, disease impact on physical function, and adverse events, which are elements of the patient experience, are difficult or often impossible to evaluate without patient input.
An important step in overcoming PRO measurement challenges is the clear articulation of measurement targets, which are dependent on hypothesized treatment effectiveness claims. In other words, it is only after we know what we want to say about the drug that we can decide what to measure, and how. This is characterized by a frequently used phrase in measurement, and the title of one of the Drug Information Association’s meetings, “Beginning with the end in mind.”38
However, outcome researchers often have little insight or influence on the sought-after drug efficacy claims, because they may not have access to early-stage data, or they may not be asked to contribute to the target product profile. Without knowledge of the sought-after claim, and thus targets of measurement, outcomes researchers are often inadequately equipped or positioned within their organizations to make measurement recommendations and perform their task successfully.
One tool that drug developers can rely on to inform an answer to the question of “what do we want to say about our drug” is a disease-specific conceptual model (Figure). Used in oncology and in other diseases to understand and document the relevant aspects of a disease, a conceptual model can help stakeholders visually conceptualize a disease in terms of its primary characteristics.39-42 As suggested by Wilson and Cleary, a conceptual model is a heuristic that classifies and links a disease state or condition to its proximal and increasingly distal health outcomes.43
In general, proximal concepts (eg, signs or symptoms of disease) tend to be simple, unidimensional, directly related to a condition and the effects of treatment, and characteristic of conversations between patients and physicians that inform treatment decisions (eg, a physician asking a patient, “how is your pain?”).44 Alternately, distal concepts, such as quality of life (QoL) and health-related QoL, and well-being, tend to be complex, multidimensional, increasingly less related to the condition and effects of treatment, and less characteristic of patient and physician conversation that informs treatment decisions.44
The Figure provides an example of a conceptual model for metastatic breast cancer. This conceptual model is not intended to be understood as a causal model. In other words, although the arrows indicate the dominant causal pathway, the absence of arrows (unidimensional or reciprocal) does not imply a lack of relationship. The conceptual model provides a list of concepts identified as relevant to a medical condition or treatment supported by empirical literature, direct feedback from key opinion leaders, and/or interviews with patients.
There are several ways to use this model to inform PRO measurement decisions. First, oncology drug developers can benefit by understanding that regulators will be more, less, or not at all inclined to positively review PRO-based efficacy claims for labeling when they are related to proximal concepts (eg, symptoms) relative to more distally supported ones, such as health-related QoL and QoL, respectively. This observation is consistent with the FDA’s position that QoL claims are inappropriate for labeling.19
It is also supported by the FDA’s record of approval between 1995 and 2010, which shows the agency approved 10 oncology label claims, of which 7 were symptom-focused and 2 were health-related QoL–focused17 (which is consistent with labels across therapeutic areas).45 However, it is unlikely that these types of assessments will yield drug label claims in the United States and, therefore, they may best be implemented in trials to support other communication vehicles.
Content validity is described as the evidence demonstrating that a PRO questionnaire comprehensively, albeit practically, measures concepts within a given domain (eg, symptoms or physical function) that are important and relevant to the target patient population, and does so in ways that respondents can understand and can provide meaningful responses for.19 Pertinent here are the first components of content validity, the importance and relevance of the concept being measured, because even before a PRO questionnaire can be selected, evaluated, or built, the researcher must know what the targets of measurement are. This is a particularly important, although often misunderstood, point: regardless of whether a sponsor chooses to build a new PRO questionnaire or to adapt an existing one for use, it is incumbent on the drug sponsor to know the target of measurement.
The evidence used to substantiate the claim that, for example, symptoms A and B and health-related QoL impacts C and D are important and are relevant to disease E, and, therefore, belong in a conceptual model, typically come from 3 data sources, including existing literature, clinical experts, and patients. In oncology, however, identifying and defining measurement concepts come with unique challenges.
For example, substantiating measurement concepts requires qualitative research data that are generated from patient concept elicitation interviews, and oncology researchers are limited by the availability of patients for these interviews, because of the rarity of the disease, the health status of participants, and/or the nuanced target patient population criteria (eg, patients with inflammatory anemia, current symptoms, “severe” fatigue, or who receive a specific treatment). Moreover, although the appropriate sample size for qualitative research purposes is frequently discussed,46 few empirical guidelines exist to inform those decisions.47
Other challenges in oncology research relate to defining concepts of measurement. For example, patients with cancer experience many disease-related and treatment-related symptoms, and distinguishing among them is difficult. Different genetic mutations that are associated with various cancer types further complicate the situation. For example, do patients with breast cancer with the BRCA1 or BRCA2 mutation have different disease-related symptoms? Understanding the similarities and differences in the phenotypic expression of select cancer types dependent on their unique genetic mutations would be beneficial for making decisions about what are the important concepts for a given cancer type.
When determining whether a PRO questionnaire can support drug labeling, the FDA relies heavily on evidence of content validity28 and, specifically, that the target concepts of measurement be clearly linked to the claim and are justifiable and substantiated with evidence.
Researchers have documented that inadequate evidence of content validity is among the primary reasons that PRO-based labeling claims have not been approved by the FDA between 2006 and 2010, with specific issues related to an unclear or nonexistent link between the claim and the measurement concept, insufficient documentation in the target patient population, and insufficient content coverage.29
It is important to recognize that although a premium has been put on patient-based qualitative data to substantiate the “appropriate and comprehensive” concepts for a given condition, the FDA also acknowledges the importance of documenting input from other sources, including the literature and clinical experts to inform questionnaire content validity.19
Given that the PRO Guidance is only a guidance as opposed to mandatory regulations, it would be of value for drug sponsors to consider these alternate approaches to generating the evidence needed for questionnaire content validity evaluations. This is particularly true in oncology, where patient-focused qualitative research is onerous and may have questionable incremental value toward the development of a reasonable measurement strategy.
Researchers can and ought to use the PRO data to evaluate efficacy for a new drug and beyond commercialization. This position is consistent with the Friends of Cancer Research patient advocacy group48; the National Comprehensive Cancer Network, whose 2012 survey indicated that among 52 oncology stakeholders, 71% characterized the use of PROs as moderately important to very important49; and the American Society of Clinical Oncology, which created a working group focused on the use of oncology PROs to improve the quality of treatment and care.50 The many, although not insurmountable,51,52 challenges toward successfully installing a PRO-based measurement strategy into an oncology trial are well-documented.40,53 These issues may be mitigated or at least anticipated only when the fundamental questions, “what do we want to say about our drug?” and “what are we going to measure to evaluate that claim?” have been thoughtfully vetted.
However, the planning and strategic components of questionnaire development, evaluation, and documentation are not often considered in drug development timelines and resources. Therefore, oncology outcomes researchers are often tasked with making measurement decisions based on insufficient evidence, which has tested the FDA’s ability to favorably evaluate those decisions and, ultimately, the PRO-based efficacy claims for US labeling.29
The FDA and, in particular, the Office of Hematology and Oncology Products, have been criticized for limiting the patient perspective in drug approval.28 This criticism is linked to the comparative paucity of PRO-based label claims in US indications for cancer drugs relative to nononcology-related labeling claims in the United States and oncology claims in Europe.16,28,54 Several complaints include impossible measurement development standards set forth in the FDA’s PRO Guidance, inconsistent and untimely measurement-based feedback provided to sponsors across FDA review divisions, and continued emphasis on survival outcomes from the industry and the FDA.55,56 The FDA has drawn attention to several initiatives designed to provide a forum for discussion with the FDA and sponsors interested in developing PRO instruments that are capable of supporting label claims.28 Among these are the Drug Development Tools Qualification Programs20 and the Clinical Outcome Assessment Compendium.57
Although the FDA initiatives provide reason for optimism, it is important for clinical researchers to remember that the FDA does not sponsor drug development or conduct clinical trials and, therefore, is not responsible for the development of PRO instruments.16 This responsibility is incumbent on drug manufacturers. It is also important to know that compared with regulations, which are understood to be mandatory, regulatory guidance characterizes the FDA’s opinion or “current thinking” on a topic and, in this respect, deviations from approaches outlined in guidances, such as the FDA’s PRO Guidance, can be acceptable to divisions, as long as they are reasonable, justified, and well-documented. In this way, it is reassuring to know that the Office of Hematology and Oncology Products and the Clinical Outcome Assessment staff are actively encouraging novel, creative, and innovative methods, procedures, and solutions.
Support for this article was provided in part by Novartis Pharmaceuticals Oncology.
Author Disclosure Statement
Dr Shields, Dr Krohe, Mr Yaworsky, Ms Mazar, and Ms Foley are consultants to Novartis. Dr Hao, Dr Mehmed, and Dr Globe are employees and stockholders of Novartis Pharmaceuticals.
1. Center for Drug Evaluation and Research. Guidance for industry: providing clinical evidence of effectiveness for human drug and biological products. May 1998. www.fda.gov/downloads/Drugs/GuidanceCompliance%20RegulatoryInformation/Guidances/UCM078749.pdf+Providing+clinical+evidence+of+effectiveness+for+human+and+bio&client=FDAgov&site=FDAgov&lr=&proxy stylesheet=FDAgov&output=xml_no_dtd&ie=UTF-8&access=p&oe=UTF-8. Accessed April 16, 2016.
2. Center for Drug Evaluation and Research. Clinical trial endpoints for the approval of non-small cell lung cancer drugs and biologics: guidance for industry. April 2015. www.fda.gov/downloads/Drugs/.../Guidances/UCM259421.pdf. Accessed April 16, 2016.
3. Brook RH, Ware JE Jr, Rogers WH, et al. Does free care improve adults’ health? Results from a randomized controlled trial. N Engl J Med. 1983;309:1426-1434.
4. Brook RH, Ware JE Jr, Davies-Avery A, et al. Overview of adult health measures fielded in Rand’s health insurance study. Med Care. 1979;17(7 suppl):iii-x, 1-131.
5. Tarlov AR, Ware JE Jr, Greenfield S, et al. The Medical Outcomes Study: an application of methods for monitoring the results of medical care. JAMA. 1989;262:925-930.
6. Johnson JR, Temple R. Food and Drug Administration requirements for approval of new anticancer drugs. Cancer Treat Rep. 1985;69:1155-1159.
7. Center for Drug Evaluation and Research. Guidance for industry: clinical trial endpoints for the approval of cancer drugs and biologics. May 2007. www.fda.gov/downloads/Drugs/.../Guidances/ucm071590.pdf. Accessed April 16, 2016.
8. Committee for Medicinal Products for Human Use; European Medicines Agency. Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products. July 27, 2005. www.ispor.org/workpaper/emea-hrql-guidance.pdf. Accessed April 16, 2016.
9. Committee for Medicinal Products for Human Use; European Medicines Agency. Reflection paper on the use of patient reported outcome (PRO) measures in oncology studies. Draft. June 17, 2014. www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2014/06/WC500168852.pdf. Accessed April 16, 2016.
10. Fayers PM, Hays RD. Don’t middle your MIDs: regression to the mean shrinks estimates of minimally important differences. Qual Life Res. 2014;23:1-4.
11. McLeod LD, Coon CD, Martin SA, et al. Interpreting patient-reported outcome results: US FDA guidance and emerging methods. Expert Rev Pharmacoecon Outcomes Res. 2011;11:163-169.
12. Hao Y. Patient-reported outcomes in support of oncology product labeling claims: regulatory context and challenges. Expert Rev Pharmacoecon Outcomes Res. 2010;10:407-420.
13. Doward LC, Gnanasakthy A, Baker MG. Patient reported outcomes: looking beyond the label claim. Health Qual Life Outcomes. 2010;8:89.
14. Mesa RA, Gotlib J, Gupta V, et al. Effect of ruxolitinib therapy on myelofibrosis-related symptoms and other patient-reported outcomes in COMFORT-I: a randomized, double-blind, placebo-controlled trial. J Clin Oncol. 2013;31:1285-1292.
15. Zagadailov E, Fine M, Shields A. Patient-reported outcomes are changing the landscape in oncology care: challenges and opportunities for payers. Am Health Drug Benefits. 2013;6:264-274.
16. Basch E. Toward patient-centered drug development in oncology. N Engl J Med. 2013;369:397-400.
17. Kitchen H, Rofail D, Caron M, Emery MP. Oncology patient-reported claims: maximising the chance for success. Ecancermedicalscience. 2011;5:212.
18. Calvert M, Kyte D, Duffy H, et al. Patient-reported outcome (PRO) assessment in clinical trials: a systematic review of guidance for trial protocol writers. PLoS One. 2014;9:e110216.
19. Center for Drug Evaluation and Research. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims. December 2009. www.fda.gov/downloads/Drugs/.../Guidances/UCM193282.pdf. Accessed April 16, 2016.
20. US Food and Drug Administration. Drug Development Tools (DDT) Qualification Programs. Updated February 27, 2015. www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/default.htm. Accessed April 16, 2016.
21. Cappelleri JC, Zou KH, Bushmakin AG, et al. Patient-Reported Outcomes: Measurement, Implementation and Interpretation. Boca Raton, FL: CRC Press; 2013.
22. Mayo NE, ed. Dictionary of Quality of Life and Health Outcomes Measurement. Milwaukee, WI: International Society for Quality of Life Research; 2015.
23. US Food and Drug Administration. Drugs@FDA Glossary of Terms. Updated February 2, 2012. www.fda.gov/Drugs/InformationOnDrugs/ucm079436.htm#L. Accessed May 5, 2015.
24. US Food and Drug Administration. Code of Federal Regulations Title 21. www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?fr=201.57. Accessed April 11, 2016.
25. Stone AA, Turkkan JS, Bachrach CA, et al, eds. The Science of Self-Report: Implications for Research and Practice. Mahwah, NJ: Lawrence Erlbaum Associates; 2000.
26. Cleeland CS, Ryan KM. Pain assessment: global use of the Brief Pain Inventory. Ann Acad Med Singapore. 1994;23:129-138.
27. US Food and Drug Administration. Clinical Outcome Assessment (COA): Glossary of Terms. www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm370262.htm. Accessed May 3, 2016.
28. Basch E, Geoghegan C, Coons SJ, et al. Patient-reported outcomes in cancer drug development and US regulatory review: perspectives from industry, the Food and Drug Administration, and the patient. JAMA Oncol. 2015;1:375-379.
29. DeMuro C, Clark M, Mordin M, et al. Reasons for rejection of patient-reported outcome label claims: a compilation based on a review of patient-reported outcome use among new molecular entities and biologic license applications, 2006-2010. Value Health. 2012;15:443-448.
30. Trask PC, Reasner DS, Bal V, et al. Patient reported endpoints and the relevance to payers and regulators: is there agreement? Presented at the Sixth Annual Patient-Reported Outcome Consortium Workshop; April 29-30, 2015; Silver Spring, MD.
31. Cleeland CS, Sloan JA; for the ASCPRO Organizing Group. Assessing the Symptoms of Cancer Using Patient-Reported Outcomes (ASCPRO): searching for standards. J Pain Symptom Manage. 2010;39:1077-1085.
32. Trotti A, Colevas AD, Setser A, Basch E. Patient-reported outcomes and the evolution of adverse event reporting in oncology. J Clin Oncol. 2007;25:5121-5127.
33. Gourgou-Bourgade S, Cameron D, Poortmans P, et al; for the Definition for the Assessment of Time-to-event Endpoints in Cancer Trials Intitiative. Guidelines for time-to-event end point definitions in breast cancer trials: results of the DATECAN initiative (Definition for the Assessment of Time-to-event Endpoints in CANcer trials). Ann Oncol. 2015;26:873-879. Erratum in: Ann Oncol. 2015;26:2505-2506.
34. Bellera CA, Penel N, Ouali M, et al. Guidelines for time-to-event end point definitions in sarcomas and gastrointestinal stromal tumors (GIST) trials: results of the DATECAN initiative (Definition for the Assessment of Time-to-event Endpoints in CANcer trials). Ann Oncol. 2015;26:865-872.
35. Mathoulin-Pelissier S, Gourgou-Bourgade S, Bonnetain F, Kramar A. Survival end point reporting in randomized cancer clinical trials: a review of major journals. J Clin Oncol. 2008;26:3721-3726.
36. Werble C. Provectus breakthrough denial is symptom of a different breakthrough at FDA. The RPM Report. June 6, 2014. www.pharmamedtechbi.com/publications/rpm-report/first-take/2014/4/provectus-breakthrough-denial-is-symptom-of-a-different-breakthrough-at-fda. Accessed May 27, 2015.
37. Dancey JE. From quality of publication to quality of care: translating trials to practice. J Natl Cancer Inst. 2010;102:670-671.
38. Drug Information Association. Beginning with the End in Mind—Study Endpoints: Targeting Patient-Centered Outcomes. March 18-20, 2014; North Bethesda, MD. www.diaglobal.org/Tools/Content.aspx?type=eopdf&file=%2fproductfiles%2f2977128%2f14023%2Epdf. Accessed May 4, 2016.
39. Bruner DW. Outcomes research in cancer symptom management trials: the Radiation Therapy Oncology Group (RTOG) conceptual model. J Natl Cancer Inst Monogr. 2007:12-15.
40. Lipscomb J, Gotay CC, Snyder CF. Patient-reported outcomes in cancer: a review of recent research and policy initiatives. CA Cancer J Clin. 2007;57:278-300.
41. Victorson DE, Anton S, Hamilton A, et al. A conceptual model of the experience of dyspnea and functional limitations in chronic obstructive pulmonary disease. Value Health. 2009;12:1018-1025.
42. Gater A, Kitchen H, Heron L, et al. Development of a conceptual model evaluating the humanistic and economic burden of Crohn’s disease: implications for patient-reported outcomes measurement and economic evaluation. Expert Rev Pharmacoecon Outcomes Res. 2015;15:643-656.
43. Wilson IB, Cleary PD. Linking clinical variables with health-related quality of life: a conceptual model of patient outcomes. JAMA. 1995;273:59-65.
44. Paty J, Shiffman S, Shields A, et al. Grasping the FDA’s PRO guidance: what the agency requires to support the selection of patient reported outcome instruments. Appl Clin Trials. 2006;15:69-77.
45. Marquis P, Caron M, Emery M-P, et al. The role of health-related quality of life data in the drug approval processes in the US and Europe: a review of guidance documents and authorizations of medicinal products from 2006 to 2010. Pharmaceut Med. 2011;25:147-160.
46. Francis JJ, Johnston M, Robertson C, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25:1229-1245.
47. Lamoureux RE, Shields A, Stokes J, et al. How many subjects are enough for symptom-focused concept elicitation studies? A retrospective analysis of saturation across twenty-six studies. Value Health. 2015;18. Abstract PRM132.
48. Friends of Cancer Research. Enhancing use of patient-centered data in regulatory decision-making. White paper. www.focr.org/enhancing-use-patient-centered-data-regulatory-decision-making. Accessed November 17, 2015.
49. Guinto JB. Data needs: assessing guidance, endpoints, and patient-reported outcomes in oncology clinical care. www.nccn.org/about/news/ebulletin/ebulletindetail.aspx?ebulletinid=128. Accessed April 11, 2016.
50. Basch E, Snyder C, McNiff K, et al. Patient-reported outcome performance measures in oncology. J Oncol Pract. 2014;10:209-211.
51. Rock EP, Kennedy DL, Furness MH, et al. Patient-reported outcomes supporting anticancer product approvals. J Clin Oncol. 2007;25:5094-5099.
52. Basch E, Abernethy AP, Mullins CD, et al. Recommendations for incorporating patient-reported outcomes into clinical comparative effectiveness research in adult oncology. J Clin Oncol. 2012;30:4249-4255.
53. Siddiqui F, Liu AK, Watkins-Bruner D, Movsas B. Patient-reported outcomes and survivorship in radiation oncology: overcoming the cons. J Clin Oncol. 2014;32:2920-2927.
54. Gnanasakthy A, Mordin M, Clark M, et al. A review of patient-reported outcome labels in the United States: 2006 to 2010. Value Health. 2012;15:437-442.
55. Fehnel S, DeMuro C, McLeod L, et al. US FDA patient-reported outcome guidance: great expectations and unintended consequences. Expert Rev Pharmacoecon Outcomes Res. 2013;13:441-446.
56. Revicki DA; for the Regulatory Issues and Patient-Reported Outcomes Task Force for the International Society for Quality of Life Research. FDA draft guidance and health-outcomes research. Lancet. 2007;369:540-542.
57. US Food and Drug Administration. Clinical Outcome Assessment Compendium. Updated January 27, 2016. www.fda.gov/Drugs/DevelopmentApprovalProcess/DevelopmentResources/ucm459231.htm. Accessed April 16, 2016.