Download the original PDF (28 pages). This page shows the full text of the PDF. The figures, some tables and the original layout are in the PDF.
Contents
om
A Multi-Pronged Strategy to Improve the Relevance, Usefulness, and Comparability of Outcomes in Clinical Research
The Green Park Collaborative is a major initiative of the Center for Medical Technology Policy
May 2, 2018 | Version 1.0
TABLE OF CONTENTS
- Lack of Incentives From Key Influencers in the Current Health Care Environment: Payers,
ABSTRACT
The lack of relevant or comparable outcomes in clinical research undermines efforts across the health care sector to make evidence-based health policy decisions and improvements to health care delivery. The use of core outcome sets (COS), agreed-upon minimum setsof outcomes that should be measured and reported in all clinical studies for related evaluations of treatments or management for a specific condition, can offer a potential solution. COS are developed with the input of relevant stakeholders, and patient engagement is a central pillar to the process of COS development in order to determine which outcomes are most relevant and meaningful to patients and other decision-makers. When COS are broadly adopted in clinical research, the anticipated result will be better clarity and alignment around relevant outcomes across multiple decision-making authorities, including regulators, payers, and health technology assessors, in addition to patients, clinicians and other health care decision makers. Furthermore, COS adoption can improve the efficiency of the research enterprise, and more quickly resolve issues of uncertainty. The Center for Medical Technology Policy (CMTP), through its Green Park Collaborative (GPC), undertook a multi-stakeholder initiative to identify ways to facilitate uptake of welldeveloped COS in clinical research. This white paper describes hurdles to COS adoption from the perspective of key stakeholders, and outlines several potential strategies to overcome these hurdles, toward the ultimate goal of improving the relevance, usefulness and comparability of research outcomes for decision-making.
FUNDING
Funding for this initiative was provided by Amgen, Genentech, Johnson & Johnson, Merck, OMERACT (Outcomes Measure in Rheumatology), Pfizer, and UCB.
CMTP WRITING AND EDITING TEAM
Rachael Moloney, MHS Research Manager and Project Lead
Jennifer Al Naber, MS, MSPH Program Manager, Green Park Collaborative
Donna Messner, PhD Senior Vice President
Sean Tunis, MD, MSc President and Chief Executive Officer
ABOUT GREEN PARK COLLABORATIVE
The Green Park Collaborative (GPC) is a major initiative of the Center for Medical Technology Policy. It works to improve clinical research by fostering collaboration and communication between drug and device developers, private and public payers, clinicians, researchers, regulators, and patients. A multistakeholder forum, GPC develops condition and technology-specific study design and policy recommendations to guide the generation of evidence needed to inform clinical, regulatory, and market access decisions.
ACKNOWLEDGEMENTS
CMTP is grateful to all the individuals who served on the project’s Steering Committee. Their generous donation of time and expertise contributed greatly to the overall success of this initiative. A special thanks to Lee Simon and Peter Tugwell of OMERACT for providing ongoing guidance over the course of the project.
CMTP would also like to acknowledge the following staff members for their dedication and hard work over the course of the past year: Janelle King, Project Assistant; Julie Simmons, Manager, Marketing and Communications; Ellen Tambor, Research Director; and Dexter Waters, Research Associate.
PART I. CONTEXT
According to Chalmers & Glasziou, “85% of research is wasted” – and the wrong choice of study outcomes (i.e., outcomes that aren’t relevant or actionable to decision-makers) can be a major contributor to this waste
a
. Outcomes reported in clinical studies for the same condition are highly variable and often do not include those most relevant to patients, clinicians, payers and others responsible for making evidence-based decisions. The lack of a common denominator in health outcomes across different sectors of health care decision-making also creates challenges for efforts to shift from traditional volume-based to value-based health care.
The use of core outcome sets (COS), agreed-upon minimum sets of outcomes that should be measured and reported in all clinical studies for related evaluations of a specific condition, can offer a potential solution
b
. For example, the Prevention of Falls Network Europe (ProFANE) COS recommends the following core outcomes: falls, fall injury, physical activity, physiological consequences, and healthrelated quality of life.
c
When COS are broadly adopted in clinical research, the result will be better clarity and alignment around relevant outcomes used across multiple decision-making authorities, including regulators, payers, and health technology assessors. Broad adoption will also create greater certainty for manufacturers, greater ability for payers to make meaningful comparative assessments of evidence, and greater assurance for patients and providers that outcomes important to them are informing critical policy decisions.
There are many different decision-making contexts related to health care that use clinical research results. As depicted in Figure 1, these include: patients as “the true end user”, making evidence based decisions with their families and doctors about treatment decisions;organizations that design and conduct systematic reviews, aggregating study results to provide a clearer picture of available evidence to help inform multiple stkaeholders; providers who struggle to standardize clinical care protocols to achieve clinically meaningful differences in patient health outcomes; and payers attempting to capture the value of therapies with variable ways to measure improvements in outcomes. Other relevant decision-making contexts include: determining the efficacy or effectiveness of health interventions, monitoring health outcomes over time, or developing clinical practice guidelines. Research results may also be used to compare the relative value of similar therapies to inform market access decisions, pricing, formulary tier assignments, or benefit design. Clinical evidence (or a lack thereof) also provides the basis for implementing coverage with evidence development
d
- and COS can shape how these or other managed entry schemes
e
are designed. These different contexts all stand to benefit from having a
a Chalmers I, Glasziou P. Avoidable waste in the production of reporting of research evidence. Lancet 2009; 374(9683): 86-9 b COMET Initiative. Available at: http://www.comet-initiative.org/.
c
Copsey B, Hopewell S, Becker C, Cameron ID, Lamb SE. Appraising the uptake and use of recommendations for a common outcome data set for clinical trials: A case study in fall injury prevention. Trials 2016; 17:31-41 d Coverage of a treatment or technology, conditional on data gathering in a registry or trial. e General term for arrangements between manufacturer and payer/provider that involve allowing market access to new drugs for which evidence is still uncertain, often characterized by performance-based agreements or risk-sharing.
“…seeing something that comes through all of this development and then lo and behold, something really, really necessary wasn’t measured at all. That’s frustrating, that’s a waste, that’s a shame.”
a
minimum set of outcomes that are consistently collected across trials and other studies for the same condition.
Throughout this paper, we refer to the organizations or individuals that utilize evidence generated by COS data to inform decisions about the use of therapies after their regulatory approval as postregulatory decision-makers (PRDMs). PRDMs include payers and purchasers, formulary committees, HTA organizations, clinical guideline developers, health care providers, and patients.
As described in more detail below, this project focuses on adoption of COS in clinical research. However, the GPC team recognized that there is a broader health system context having potentially broader implications for COS. The team therefore felt that COS uptake should not be discussed in a vacuum; i.e., only as a tool of clinical research to inform clinical decisions. In particular, since the fundamental definition of “value” in health care is determined by the improvement in outcomes relative to the cost of care, the articulation of outcomes meaningful to patients and other stakeholders is key to defining “value” for any intervention in health care. For this conception of value to be operationalized coherently (in, for example, alternative payment models or value-based payment schemes generally), a minimum set of consistently collected patient/stakeholder-important outcomes would be a highly useful resource to help connect patient-informed definitions of value to quality of care.
In these and other ways, promoting adoption of well-developed COS is key to achieving clinical research that is more efficient and more informative for decision-making by regulators and PRDMs. Even so, while COS have been in existence for years in many therapeutic areas, and many groups continue to develop and publish new COS, the actual adoption of these COS in clinical research has been inconsistent.3-8
What are the barriers to adoption of COS in clinical research, and what can be done to promote their use? To address the issue of COS adoption, GPC: 1) convened an expert steering group composed of key stakeholders (Appendix A); 2) conducted key informant interviews with medical policy and research leaders able to speak to barriers and facilitators of COS adoption throughout the health system; and 3) with input from these groups, held a one-day, in-person multi-stakeholder meeting in Baltimore, Maryland, to agree on current challenges and brainstorm possible solutions (Appendix B). As noted above, though the project aim focuses on adoption of COS in clinical research, the GPC team recognizes the importance of including diverse stakeholder perspectives to add depth to contextual understanding and enhance the quality of discussion. This white paper describes key hurdles to COS adoption, from the perspective of diverse stakeholders, and outlines several potential strategies to overcome these hurdles, toward the ultimate goal of improving the relevance, usefulness and comparability of outcomes in clinical research.
“People are either not understanding or are not listening –[to the fact that] one of the major failings of clinical research is the incongruence and inconsistency of what we measure, and it’s just astounding.”
Figure 1. Current Challenges for PRDMs Related to Outcomes Reported in Research
PART II. MAJOR CHALLENGES AND CONSIDERATIONS
Decision-making at the individual, health system and population levels, as well as decisions about market access, all stand to benefit from broader adoption of COS (Figure 2). Still, there are numerous hurdles and issues related to COS and their adoption across different areas of health care and research.
1. LACK OF AWARENESS (OUTSIDE ACADEMIC CIRCLES) OF COS FOR RESEARCH
The situation at hand is that COS intended for research have proliferated in many therapeutic areas, with many stakeholders and resources involved in the process of developing them – yet clinical researchers are not consistently building COS into their studies. The level of awareness among clinical researchers may vary, but there is also lack of awareness of COS among PRDM organizations. Though we did not systematically seek to explain why this lack of awareness persists, we hypothesize that there is still a vast gulf between researchers and policy makers, and that COS developers, who are mostly academics and publish in research journals (as opposed to policy journals), do not really know how best to connect with or deliver their work to the policy world. The lack of awareness of COS among PRDMs translates into a lack of pull, or demand from PRDMs, that would incentivize researchers to use COS; if it were clear that PRDMs preferred to see clinical studies incorporating COS that reflect their concerns and those of patients, clinical researchers would be more likely to respond in the design of their studies. It follows that PRDMs need to express these preferences to promote more consistent adoption of COS into clinical studies. However, awareness of COS among PRDMs must be prioritized and achieved before they can be asked to formally acknowledge or express preferences for the use of COS and communicate these preferences to clinical researchers. These organizations and their constituencies need an opportunity to become familiar with COS and recognize the potential benefit of COS adoption in support of their own business purposes or missions, instead of viewing COS adoption as a compromise with other groups. “There’s a huge amount of intellectual horsepower…that the clinical day to day world is unaware of. [We’re] not taking advantage of the deep work that the pharma companies and the trade groups and academic departments have done, which is unfortunate. It’s the same human body.”
Figure 2. Introduction to Core Outcome Sets
2. GROWING NUMBER OF SIMILAR BUT SILOED INITIATIVES
A growing list of parallel efforts are standardizing outcomes or outcome measures across different sectors of the health care field, from research to quality improvement to performance measurement (Figure 3). Organizations that have increased their level of activity in defining valid, reliable and meaningful measures of quality and patient outcomes include: National Quality Forum (NQF), International Consortium for Health Outcomes Measurement (ICHOM), the Centers for Medicare and Medicaid Services (CMS), the Patient-Centered Outcomes Research Institute (PCORI) and others. The fact that there is a growing trend toward outcomes standardization across health care sectors underscores that many different stakeholder organizations perceive value in doing so. While not all the outcomes selected for these different purposes readily translate between contexts, these work streams are moving forward without full attention to ensuring coordination wherever possible.
Furthermore, many stakeholder participants fear that the rise of alternative payment models and other risk-sharing arrangements are creating an environment tending towards more fragmentation of outcome measures into siloes created by professional societies and organizations engaged in negotiations over quality metrics and payment. By tying these new arrangements to a range of diverse outcomes, even more confusion can arise. This creates a flurry of mixed signals and conflicting standards within which researchers and product developers must navigate even more laboriously.
The degree of overlap and relationship between COS and other outcomes standards in different fields (such as ICHOM standard sets or CMS core quality measures) is not yet entirely clear. Without recognizing the existence and relationship of these many efforts, we may find ourselves with multiple segregated and inconsistent standards, instead of a paradigm in which research outcomes relevant to multiple stakeholders and care settings are harmonized to the extent possible (recognizing that direct translation of outcomes across contexts will often be infeasible, but conscious assessment may lead to opportunities for harmonization or better understanding of the relationships between similar outcomes intended for different contexts). In clinical research alone, there can be multiple COS that appear to meet minimum standards. But how do researchers know which one to use? For example, there are over half a dozen different groups that have worked to develop core outcome domains in low back pain, with varying degrees of overlap between any two COS (Appendix C). “There’s a lot of duplication already going on in several big organizations. And it’s amazing to hear about that and is also a little bit frightening that so many people are doing and hearing it the first time here when we’re sitting next to each other. It makes me think that if people don’t have cross talk and cross pollination, that we’re going to come out with three different standards and seven different ways of doing things, which, in the end, may not advance the field….”
Figure 3. Key Players in the Current Outcomes Landscape
3. LACK OF INCENTIVES FROM KEY INFLUENCERS IN THE CURRENT HEALTH CARE
ENVIRONMENT: PAYERS, RESEARCH FUNDERS, HEALTH TECHNOLOGY ASSESSMENT,
AND SYSTEMATIC REVIEW ORGANIZATIONS
So long as key stakeholder organizations who may influence the funding or dissemination of research results (e.g. payers, HTA organizations, journal editors, clinical guideline developers, research funders, industry trade associations) stay silent on what outcomes they want to see from researchers and manufacturers, efforts to promote COS adoption among researchers will lack real impact. These groups play a critical role in the health care and research environment, often serving as advisors on study design (e.g. through early scientific advice), and later as gatekeepers to public dissemination of research results (affecting the influence of certain results in important policy decisions that impact what and how health care and therapies are delivered to the public). Several European organizations and agencies have publicly endorsed COS and direct researchers to the COMET Initiative when preparing proposals and submissions.
f
However, a greater demand for COS from PRDM voices would garner attention from researchers and product developers alike -- particularly in the U.S., where the private health care landscape fosters many disparate and sometimes conflicting evidence standards across different groups, as opposed to a clear directive from a single authoritative agency.
For example, signals from stakeholder groups, such as public and private research funders, journal publishers and evidence assessors, could contribute to alignment on the academic front. Whereas project participants noted that researchers in industry would likely see greater value in using COS if they believe payers will favor them. Value-based payment and alternative payment models could potentially offer conditions to demonstrate the feasibility of payment and accountability structures that incentivize collection of certain provider and patient-reported outcomes from existing COS.
As mentioned previously, however, before the payer community (or any other PRDM community) can endorse and ask for COS from researchers, it first needs greater awareness of the potential role of COS in clinical research and whether existing COS contain some outcomes that capture essential information relevant to their decision-making needs. Furthermore, payers need to know that it is possible and feasible to capture a COS for a particular episode or population, within the parameters of available data and resources. There is still a disconnect between knowing that measures to capture COS exist, and believing that data on COS can reliably be captured at scale.
4. LACK OF CONFIDENCE AND FAMILIARITY WITH THE COS DEVELOPMENT METHODOLOGY
f www.comet-initiative.org/cosuptake
“It is going to be total measurement chaos because every professional society is proposing a payment model for their tiny little slice of the body with its own measures and coming up with them out of nowhere and nobody is telling them [researchers], no, you must be using this core outcome set.”
“Because there is no quality framework to assess a COS, you can still do what you like, really.”
The quote above reflects the current perception of many of the COS project participants. Partly because of the general lack of awareness of COS, there is also limited familiarity and confidence in current approaches for developing COS. Granted, the methodology for COS development and how stakeholder consensus is reached does vary somewhat from effort to effort. While the prevailing perception is a “total free-for-all” of COS development without standards, there are in fact several groups, including OMERACT and COMET, that have dedicated substantial attention to developing and articulating minimum quality standards for COS development.8,9 It is true that there is no singular, broadly accepted, formal assessment framework with which a particular COS can be assessed to ensure confidence in its value, or between which two similar COS can be compared for quality and relevance. But the building blocks for such a framework are already in place. Greater effort is needed to disseminate and promote awareness of their existence.
Stakeholder engagement in COS is also sometimes hard to define and assess. Whether COS include meaningful patient engagement, as well as other stakeholder perspectives, and how those perspectives are weighted and incorporated into the final core set, can be difficult to ascertain without a guide or standard against which to compare. This is important because patients sometimes have different priorities or perspectives than clinical experts on what outcomes matter the most
g
:
As one participant noted, for some conditions clincians must rely solely on the views of patients to ascertain and treat a disease:
The fact that meaningful patient engagement and input is a cornerstone of COS development methodology should be the focal point of messaging and be made very clear to potential adopters of COS who are interested in capturing patient definitions of value.
5. SELECTING THE APPROPRIATE MEASURES AND INSTRUMENTSh
Much work and multi-stakeholder input goes into the development of COS. However, for a COS to be successfully implemented, researchers need to have the tools to measure the outcomes in a scientifically valid and consistent manner. The existence and use of multiple instruments by researchers to measure a given outcome remains a major challenge to aligning and standardizing outcomes in a way
g According to Berglas et al, in a series of CADTH drug assessments, the outcomes included in manufacturerinitiated trials differed from patient perspectives on the most desired outcomes by as much as 50%. Berglas S, Jutai L, MacKean G, Weeks L. Patients’ perspectives can be integrated in health technology assessments: an exploratory analysis of CADTH Common Drug Review. BMC Research Involvement and Engagement 2016; 2:21. h See Appendix D for terminology,
“We often hear from patient groups that there may be a disconnect between what the clinical community views to be a success and what the patients do.”
“We cannot see the disease called acute gout. We can’t see that. It’s not anywhere out there for us. It’s what the patients tell us. They are what defines the disease because we cannot see the disease.””
that can help decision makers. In many therapeutic areas, there may be numerous instruments, validated or not, in use for particular outcomes. On a new, advanced prosthetic technology:
Certain instruments are also proprietary, and the added monetary costs for using proprietary instruments can be frustrating for groups who are trying their best to measure COS.
To complicate things further, across validated approaches to measures, there may be limited understanding of differences between measurement instruments. While there are several groups working to guide measurement instrument selection, among them the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN)
i
, awareness among researchers of available guidance, and whether it conflicts with recommendations from other groups, is still limited.
6. GENERAL CONCERNS ABOUT THE USE OF COS FOR CLINICAL RESEARCH
Early in the project, several key informant interview participants expressed concerns that standardization comes at the cost of innovation:
Concerns that the promotion and uptake of COS may deter the development of better measures not currently included in a core set seem to be a serious hurdle for both researchers and other groups in the outcomes landscape to get behind COS uptake. Several participants asked to know what mechanisms will remain in place to allow for changes or updates to COS. As health technologies evolve beyond what is currently captured by COS, will core outcomes and their measures need to evolve?
Other concerns arose from researchers themselves, concerned about having the flexibility they need to select the outcome measures that they believe best reflect the specific questions at issue in individual trials. If an investigator must power a study to meaningfully include additional outcomes from the COS, there is concern that the study may become prohibitively large and grow in cost and complexity.
i https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-016-1555-2
“You have this really science-fictiony device, there’s a delay in getting it to patients, but of the studies, there was something like 11 studies that measured 72 different outcome measures, and none of them were validated in the correct population.”
“There were so many measures out there. It was overwhelming to try to codify what should be the main measure, or couple of measures…it’s a total free for all.”
“Constancy may be very good for variability, but it’s not very good for technology change.”
“I don’t think we’re going to move forward being collectors. We’ll move forward being curators.”
A discussion point raised by one informant was whether investigators in clinical development would have to measure every outcome of a COS in a single trial, as opposed to including some outcomes in later real-world studies. This reflects the struggle of balancing important outcome collection with additional secondary outcomes relevant to studies’ specific aims.
The question of who reports the data and with how much of a burden is also part of this particular debate. Whether the COS are patient reported, clinician reported, laboratory reported, etc. can have implications for feasibility. Over-collection of outcomes by researchers can encumber progress. In addition to out-of-pocket costs, studies that are over-burdened with data collection may have more difficulty getting patients to sign up and take longer to complete. Health systems also have limited capacity to collect outcomes of importance.
Additional concerns about COS in general include:
- conflicting opinions regarding whether to focus on condition-specific COS, or to focus on developing and promoting standardized outcomes and measures that take a more global, whole-person view of patients;
- unaddressed concerns about the cost of COS development, and more importantly, the typically long duration for new COS to be developed, measures validated, and made available to researchers – COS development requires greater streamlining to become usable in a workable timeline;
- the perception for some industry stakeholders that use of COS in clinical development would hamper the ability to report on selected outcomes offering a potential competitive advantage as compared to other products; and
- questions or concerns regarding what organization (in existence or yet to be created) might need to be the governing entity to ensure COS quality and updates to standards.
PART III. POTENTIAL SOLUTIONS
With the value of COS agreed upon and many of the challenges of COS uptake identified, stakeholder meeting participants broke into small groups to brainstorm solutions. The breakout questions and topics were identified through previous key informant interviews, in consultation with the project steering group. The breakout topics mirrored the challenges and barriers discussed throughout the early sessions of the meeting, and included: creating incentives, improving measurement, and standardizing COS development methodology. Figure 4 shows a graphic representation of the multi-stakeholder dialogue specific to brainstorming potential strategies to promote COS uptake in research. The following content is based on that dialogue, and provides greater detail.
Figure 4. Strategies to Facilitate Broad Uptake of COS in Clinical Research
CHALLENGE: LACK OF AWARENESS (OUTSIDE ACADEMIC CIRCLES) OF COS FOR RESEARCH
Solution: Promote Awareness of COS and their benefits among broader stakeholder communities A COS awareness campaign, whether lead by an existing group or a new multi-stakeholder consortium, would be useful to target messages to key stakeholder groups emphasizing the potential of COS to improve relevance, usefulness and comparability of research results for decision-making.Materials should be written or produced to help different groups of decision makers – including payers and other PRDMs – understand why they should be asking for COS, by emphasizing how evidence on COS will improve the quality and ease of decision-making. Such a campaign should also identify important annual meetings of PRDM stakeholders communities, such as Guidelines International Network (G-I-N), Health Technology Assessment International (HTAi) or Academy of Managed Care Pharmacy (AMCP). Similarly, targeted messaging and outreach to promote collection and reporting of COS should speak to different groups of researchers, from academic researchers, who have their own motivators and incentive structures, to drug developers in pharma, who operate within a different set of motivators and market incentives.
In addition, awareness efforts should:
- have clear, concise, terminology that builds upon existing knowledge, e.g. from the Core Outcome Measures in Effectiveness Trials (COMET)
j
Initiative or the Outcome Measures in Rheumatology (OMERACT)
[This table or figure is not reproduced here. See page 17 of the PDF.]
;
- describe how COS can improve overall applicability and usefulness of clinical research results;
- describe how COS and other standardized outcome sets not labeled as COS (but used for research, clinical care, value, and quality) are related and how/why they differ;
- educate organizations on the role of patients in COS development, particularly the fact that the focus on patient input in COS development will promote the consistent inclusion of patient values in clinical research and beyond;
- recruit and organize expert and stakeholder champions to present and promote awareness of COS at annual meetings and conferences being planned by key organizations; and
- allay some of the general concerns about COS (Part II), for example by explaining how COS are not restrictive at the expense of measurement of other context-specific outcomes, or explaining ways that COS can be updated and modified over the course of time and advances in health technology.
CHALLENGE: GROWING NUMBER OF SIMILAR BUT SILOED INITIATIVES
Solution: Map the Environment An environmental scan to identify, describe and “map” the landscape of related initiatives in both the outcomes and measure standardization space could help to uncover areas of overlapping efforts, gaps
[This table or figure is not reproduced here. See page 17 of the PDF.]
For example: what is the difference between measure and instrument? What is the relationships between a performance measure and a quality measure? It will be important to have the terminology precisely defined in order to organize and align efforts correctly.
“Obviously in order to have consensus, you need to see what’s out there, right?”
or needs, shared values and methodological standards. The very act of formally calling out related initiatives would also facilitate awareness and understanding between the many players working to create standards. It could help create opportunities for outreach and the forging of new connections and collaborations between heretofore siloed groups and avoid future missed opportunities to converge toward shared standards and values.
Solution: Form a Network of Stakeholder Organizations Linking organizations involved in developing COS could help to build consensus on standards and best practices in COS development, reduce variability and duplication of efforts, and create a higher profile for the work being done.
More specifically, a network might offer the following:
- a searchable online index of organizations, with points of contact and descriptions of work for each organization, that could be accessed and queried by interested parties;
- an index of useful links, such as the COMET Initiative database
m
, to direct interested stakeholders to existing resources with additional information on available COS;
- an index of useful links to publicly accessible repositories of measures and measurement instruments, such as PROMIS
n
or the AHRQ National Quality Measures Clearinghouse
o
; o promote such a repository (whether stand-alone or embedded in an existing index such as COMET) as a place for researchers to look first when designing their study;
- a central repository of resources and communications across committed organizations; and
- a platform to promote: o better awareness of COS; o collaboration across parallel efforts to develop and promote the use of outcomes that are relevant and comparable across research and other health care initiatives; o broader stakeholder understanding of, and participation in, efforts to identify and support holistic measures of disease impact; and o consensus-building around standards for developing generalized (non-disease specific) core outcome sets.
The network would ideally be international in scope and coordinated from a central organizing hub which would curate the database, manage resources and communications, put questions to the group for resolution (facilitating discussions among network members), plan periodic in-person group conferences, and develop strategies for external communications and promotion.
Finally, the network could serve as a vessel for disseminating stakeholder and expert-informed guidance for COS development, both new and existing. If new guidance, or updates to existing guidance (e.g. building upon that of OMERACT9 and COMET8), is written within the context of the broader outcomes standardization landscape, such guidance would help to reduce variation in future outcomes standards, improve the transparency and reproducibility of high quality COS development, and foster greater confidence, support, and adoption of COS. Ultimately, methodological guidance might cover the entire process of COS development, from project inception and refinement, to evaluation, dissemination and implementation.
m http://www.comet-initiative.org/ n https://commonfund.nih.gov/promis/index o https://www.ahrq.gov/cpi/about/otherwebsites/qualitymeasures.ahrq.gov/index.html
Other ideas for content in future guidance include:
- A COS Development Checklist, which also highlights multiple opportunities throughout the COS development process for patient and stakeholder feedback
- Links out to sites (such as COMET) where existing COS work might be searched and identified, as well as other health care fields with relevant work, ongoing or in development
- Patient engagement rubric specifically tailored to COS, to help developers formally describe their engagement plan over the course of developing their COS
- Clinical trial protocol templates updated with COS, that meet industry and regulatory standards
- Other templates and tools specifically for embedding implementation and evaluation into COS development.
Over time, identifying and measuring COS in research can become more streamlined, less burdensome, and more successful for both clinical development and academic researchers.
CHALLENGE: LACK OF INCENTIVES FROM KEY INFLUENCERS IN THE CURRENT HEALTH CARE ENVIRONMENT
A number of decision-makers in health care coverage, delivery, publication, and clinical practice guidelines can help to promote the uptake of COS in clinical research and medical product development. The following sections describe how key groups can incentivize the uptake and reporting of COS. Each section is based on the premise that PRDMs, by telling researchers what outcomes are relevant and actionable for their decision-making purposes (e.g. coverage, formulary design, provider performance, etc.) can better reinforce efforts to promote COS uptake.
Solution: A Role for Payers Traditionally, health plans have provided generalized guidance on the type of evidence needed for decision-making (for example, evidence that a product is “reasonable and necessary” for patient care, or that it is “medically necessary” given a specific intended use and context of care). Since this evidence can take many forms, health plans have not specifically dictated outcomes and measures they would like to see. However, if a standard set of core outcomes and measures were used in the development of products having the same intended uses, it would be easier for health plans to understand the comparative benefits and risks of each product. Moreover, if payers are given the opportunity to provide input on COS, then they could have greater assurance of having relevant information for decision-making when reviewing products developed with those COS. Finally, manufacturers look for the signal to confirm a true shift in purchasing and reimbursement. Endorsement and prioritization of COS in reimbursement practices and policies around products and technologies may provide that signal.
Additionally, a growing segment of health care reimbursement is based on alternative payment models (APMs), meaning alternatives to traditional fee-for-service payment that are designed to differentially reward health care providers based on the quality of care that is delivered to their patients. A current “The ideal signal would be if CMS said okay, we are going to pay for arthritis drugs based on these measures that signal when somebody is developing a new RA drug is thinking, oh five years from now I’m going to be paid based on whether or not I hit this marker and there is a value chain all the way down so the developers are on the same wavelength.”
challenge in the development and negotiation of APMs is that the outcomes on which these contracts are based must be accessible and measurable. The variability in outcomes used in clinical research studies and collected in electronic health records can complicate the development of workable APMs. Moreover, this variability can lead to different frameworks for reimbursement (and accordingly different underlying assessments of value) from payer to payer, therapy to therapy, and health system to health system.Greater consistency of outcomes across the spectrum of research, from medical product development to negotiation of APMs, may help to improve consistency – and consistent operating definitions of value – in the development and application of APMs. The same reasoning applies to outcome-based contracts, a negotiated mechanism for outcomes-based differential payment for medical products negotiated between health plans and medical product companies.
Solution: A Role for Providers: COS may also offer an opportunity to assure that patient-important outcomes are included in the assessment of provider care. Providers participating in APMs may advocate for greater capacity and infrastructure to report on outcomes they feel are better representative of the quality of the care they provide to their patients. Among the most meaningful measures of health care quality and patient experience are patient reported outcomes, such as those stipulated in robustly developed, patientcentered, COS. Providers can play a role in shaping the care delivery sector’s readiness and ability to measure COS by working with electronic health record (EHR) vendors to identify and request updates to EHR infrastructure.
Solution: A Role for Research Funders and Journal Editors Research funders and clinical trial registries could develop language that encourages the use of COS as appropriate, and encourages applicants to describe why COS are not incorporated in their submissions. The goal is to familiarize researchers with the existence of COS and where to find them, and get into the habit of COS in their own studies, without the pressure of penalty (if they do not). There are a growing number of researchers in the UK who have experienced this type of “soft encouragement” through NIHR
p
proposals, which direct researchers to search the COMET database for established COS. In the current environment, while variability still exists and the COS development methodology is still maturing, a soft approach is one that both private and federal research funders can take, without fear of unintended consequences due to their own uncertainty with COS.
Journal editors and research funders can also take a similar approach for publications or proposals for new COS. There is a precedent in clinical trial reporting and funding protocol submission requirements, in which both now expect and require systematic reviews of existing literature, followed by discussion of how reported or proposed work contributes to remaining evidence needs or gaps in understanding. Therefore, such a strategy if promoted within the context of mitigating multiple core or standardized outcome sets, would not be far out of the realm of research norm or cultural familiarity.
Federal research funders may also take additional steps to facilitate organizational culture change around COS. This means addressing funding incentives at multiple levels: individual, departmental, and institutional. Positively encouraging collection and reporting of COS that is followed by the introduction of real consequences to individual and organizational qualifications for future funding is one approach brainstormed by stakeholders at the GPC COS in person meeting.
p
“Where established Core Outcomes exist [on comet-initiative.org] they should be included amongst the list of outcomes unless there is good reason to do otherwise.” Guidance on Applications. National Institute for Health Research. https://www.nihr.ac.uk/funding-and-support/documents/current-funding-opportunities/hta/17_68cb.pdf
Solution: A Role for Evidence Assessors Several stakeholder groups may facilitate uptake of COS by influencing evidence hierarchies and standards that are used to assess and evaluate the quality of evidence for inclusion in systematic reviews, clinical guidelines, and health technology assessments. HTA organizations may want to emphasize their growing need for consistent outcomes across an increasingly “product life cycle” approach to HTA and decision-making. Multiple decisions are made and revisited over the course of time during which a product enters and remains on the market. Though approaches vary by country or organizations, many HTA organizations have multiple avenues to connect with product developers and communicate their preference for COS. These may include, for example:
- updating methodological standards to include COS use;
- specifying how inclusion or exclusion of COS in available studies influenced decision-making in final reports; and
- bringing COS to the attention of product developers via early scientific advice discussions.
These are all ways to send a clear message to researchers and product developers early in the evidence development process, where there is still opportunity to revise studies and shape upcoming evidence to be more relevant to decision-making.
CHALLENGE: LACK OF CONFIDENCE AND FAMILIARITY WITH THE COS DEVELOPMENT METHODOLOGY
Solution: Build Consensus Around Minimum Standards for COS Development Building upon the work started and recently published by Kirkham et al (2017)8 and Boers et al (2018)9, develop minimum standards for COS development into a stakeholder-vetted, consensus-based COS quality assessment framework.
COS assessment should include an evaluation of the extent to which the COS is being adopted, and include ongoing opportunities for those piloting or adopting COS to provide feedback for the purposeof re-evaluating or revising core domains and measures. This allows the dynamic nature of a COS to be built in to the process, and makes the potential for updates a standard part of COS development.
CHALLENGE: COMPLEXITY OF MEASUREMENT
Solution: Improve Understanding and Consensus Around What Constitute “Good” Measures Build upon current work by COSMIN
q
and others to develop greater consensus around criteria for “good” measures and measurement instrument selection. Consider existing standards that other
“Adding the word ‘update’ because…you can feedback and not do anything. Update means you’re going to do something about whatever feedback.”
“A lot of the [measurement] instruments were developed 15 or 20 years ago when people were not publishing a study on each property of an instrument…Things have evolved. And so has methodology.”
organizations use for selecting measures
r
, explore areas of overlap and generate better understanding of the contextual differences for which measurements and instruments are used.
The objective is to build upon existing work for measuring COS in research by supplementing current knowledge with greater insight from related health care sectors. This will promote awareness and understanding of contextual differences in measurement needs, either through guidance on measure selection, minimum standards for evaluating the quality of instruments, or multi-stakeholder consensus around principles for “good” measurement.
DISCUSSION
As noted above, the direct benefits of broad COS adoption include: less variability in research results (enabling aggregation and comparison of evidence from multiple studies), and study outcomes that are consistently meaningful for patients, clinicians, payers, and other decision makers across multiple health policy and health care decision contexts. PRDMs and other proponents of COS should also stay in tune with legislative changes that influence regulatory policies related to acceptable outcomes for regulatory decision-making, patient-informed drug development, and pharmaceutical information exchange. For example, future legislation may increase communication between manufacturers and payers prior to regulatory drug approval. This would enable benefit designs to occur prior to products reaching the market. If COS are part of those communications, this movement could enable consistency in evidence before and after regulatory approval.
Ultimately, COS adoption could even lead to faster patient access to therapies. As research reporting on COS proliferates, measures for those outcomes will be used repeatedly, leading to greater understanding of measurement effects and how different measures perform in different patient groups. This paper points to several potential solutions that can be undertaken by PRDMs to encourage and promote broader adoption of COS, with the ultimate goal of more relevant, useful and comparable research results. Health technology manufacturers will also benefit from clearer signals from PRDMs regarding evidence expectations and how evidence is assessed to determine value. They can use this information to better design studies to meet decision maker needs.10 Adoption of COS may help to simplify the design of RCTs, and, with greater standardization in outcomes, COS adoption may diminish the need for additional head-to-head trials post-marketing by facilitating indirect treatment comparisons. Thus, there is potential to lower development costs, leaving more room for innovation.
CONCLUSION
The Center for Medical Technology Policy is committed to promoting awareness and understanding of COS, and bringing together siloed groups to increase alignment. Next steps will involve promoting greater awareness of COS, broader consensus and dissemination around standards for COS development methodology, and connecting siloed organizations with shared goals of improving research outcomes for decision-making across health care.
r For example: What are NQF’s criteria for “good” measures? What are PatientsLikeMe principles for patient informed measurement principles?
“Clinical trials are only as relevant as their outcomes.”
-Tugwell P, Boers M. J Rheumatol 1993;20(3):528-30
REFERENCES
- Gonzales JA, Gritz DC, Channa R, Quinto GG, Kim A, Chuck RS. Non-steroidal anti-inflammatory drugs versus corticosteroids for controlling inflammation after uncomplicated cataract surgery (protocol). Cochrane Database of Systematic Reviews 2013, Issue 5. Art. No.: CD010516. DOI: 10.1002/14651858.CD010516.; Juthani VV, Clearfield E, Chuck RS. Non-steroidal antiinflammatory drugs versus corticosteroids for controlling inflammation after uncomplicated cataract surgery. Cochrane Database of Systematic Reviews 2017, Issue 7. Art. No.: CD010516. DOI: 10.1002/14651858.CD010516.pub2.
- Juthani VV, Clearfield E, Chuck RS. Non-steroidal anti-inflammatory drugs versus corticosteroids for controlling inflammation after uncomplicated cataract surgery. Cochrane Database of Systematic Reviews 2017, Issue 7. Art. No.: CD010516. DOI: 10.1002/14651858.CD010516.pub2.
- Copsey B, Hopewell S, Becker C, Cameron ID, Lamb SE. Appraising the uptake and use of recommendations for a common outcome data set for clinical trials: A case study in fall injury prevention. Trials. 2016;17:31-41.
- Baustista-Molano W, Navarro-Compan V, Landewe RBM, Boers M, Kirkham JJ, van der Heijde D. How well are the ASAS/OMERACT core outcome sets for ankylosing spondylitis implemented in randomized clinical trials? A systematic literature review. Clin Rheumatol. 2014;33:1313-1322.
- Kirkham JJ, Clarke M, Williamson PR. A methodological approach for assessing the uptake of core outcome sets using ClinicalTrials.gov: findings from a review of randomized controlled trials of rheumatoid arthritis. BMS 2017; 357:j2262.
- Thomas K. EASI does it: A comparison of four eczema severity scales. British J Dermatol. 2015;173:316-317.
- Howell D, Molloy S, Wilkinson K, et al. Patient-reported outcomes in routine cancer clinical practice: A scoping review of use, impact on health outcomes, and implementation factors. Annals of Oncology. 2015;26:1846-1858.
- Kirkham JJ, Davis K, Altman DG, et al. Core Outcome Set-STAndards for Development: The COS- STAD recommendations. PLoS Med 2017;14(11):e1002447.
- Boers M, Kirwan JR, Tugwell P, et al. The OMERACT Handbook. Published by OMERACT. Updated version: January 19, 2018. https://omeract.org/resources.
- Miller M. Understanding evidence on the value of medical technologies. Prepared for AdvaMed. https://www.advamed.org/sites/default/files/resource/understanding_evidence_on_the_value _of_medical_technologies_05122017.pdf. Published May 2017. Accessed January 2018.
APPENDIX A
COS PROJECT STEERING COMMITTEE
Naomi Aronson, PhD BCBSA TEC Sandra Arthurs, MS, BME Kaiser Permanente Elise Berliner, PhD AHRQ
Daniel Canos, PhD, MPH CMS Phil Conoghan, MD, PhD University of Leeds Tammy Clifford, PhD CADTH
David Cella, PhD Northwestern University Joseph Cook, PhD, JD Pfizer Chris Gale, PhD, MBBS Imperial College London
Angus Gunn UCB Pharma Ltd. Allison Hoffman, PhD FDA David Lansky, PhD ICHOM
Jeff Lemay, JD Jacqueline Vanderpuye-Orgle, PhD Amgen Amye Leong, MBA Healthy Motivation Leona Markson, ScD Merck & Co.
Michelle Mujoomdar EUNetHTA Sally Okun, RN PatientsLikeMe Dan Ollendorf, PhD ICER
Beverly Shea, PhD University of Ottawa Lee S. Simon, MD SDG, LLC Peter Tugwell, MD, MSc University of Ottawa
Evelyn P. Whitlock, MD, PhD PCORI Rebecca Williams, PharmD, MPH NIH Paula Williamson, Prof University of Liverpool
Kimberly Woodruff, PharmD, PhD Janssen Scientific Deborah Zarin, MD NIH Katy Harrison, Prof NICE
APPENDIX B
COS MULTI-STAKEHOLDER MEETING PARTICIPANT LIST
Dorcas Beaton, PhD, MSc St. Michael's Hospital, Keenan Research Centre, Li Ka Shing Knowledge Institute; University of Toronto, Institute of Health Policy, Management, and Evaluation
Elise Berliner, PhD Agency for Healthcare Research and Quality (AHRQ), Center for Outcomes and Evidence Assessment, Technology Assessment Program
Clifton (Bing) O. Bingham III, MD Johns Hopkins Arthritis Center, Initiative on Methods, Measurements, and Pain Assessment in Clinical Trials (IMMPACT)
Diana Brixner, PhD, RPh, FAMCP University of Utah, Pharmacotherapy Outcomes Research Center (PORC); Academy of Managed Care Providers (AMCP)
Helen Burstin, MD, MPH, FACP National Quality Forum (NQF); Council of Medical Specialty Societies, EVP & CEO (as of 1/1/18)
Daniel Canos, PhD Centers for Medicaid and Medicare Services (CMS), Coverage and Analysis Group
David Cella, PhD National Institutes of Health, Patient Reported Outcomes Measurement Information System (NIH-PROMIS)
Tammy Clifford, PhD Canadian Agency for Drugs and Technologies in Health (CADTH), Evidence Standards
Deborah (Deb) Collyar Patient Advocates in Research (PAIR) Robert (Bob) Collyar Patient Advocates in Research (PAIR); Institute for Clinical and Economic Review (ICER)
Joseph (Joe) Cook, JD, PhD Pfizer, Global Health & Value
Selena Daniels, PharmD, MS US Food and Drug Administration (FDA), Center for Drug Evaluation and Research (CDER), Clinical Outcome Assessments (COA), Office of New Drugs
Reena Duseja, MD, MS Centers for Medicaid and Medicare Services (CMS), Quality Measurement and Value-Based Incentives Group, Division of Quality Measurement
Christopher Gale, PhD, MBBS Imperial College London; Core Outcomes in Neonatology (COIN)
Cynthia (Cyndi) Grossman, PhD FasterCures, Science of Patient Input
Angus Gunn, MBA UCB Pharma., Ltd.
Kathryn (Katy) Harrison, Prof. National Institute for Health and Care Excellence (NICE)
David Lansky, PhD International Consortium for Health Outcomes Measurement (ICHOM)
Jeff Lemay, JD Amgen, US Innovation Policy and Research Amye Leong, MBA Healthy Motivation; Outcomes Measures in Rheumatology (OMERACT)
Elizabeth (Liz) Loughren, MPH Kaiser Permanente, Southern California Permanente Medical Group, Medical Technology Assessment and Guidelines
Leona Markson, MHS, ScD Merck, Global Health Outcomes
Robert (Bob) McDonough, MD, JD, MPP Aetna, Clinical Policy Research and Development
Gyasi Moscou-Jackson, PhD, MHS, RN Patient Centered Outcomes Research Institute (PCORI), Healthcare Delivery and Disparities Research
Matthew Moyer, MS, MBA, PMP Merck, Clinical Supply Technology
Sally Okun, RN, MMHS PatientsLikeMe, Advocacy, Policy and Patient Safety
Dan Ollendorf, PhD Institute for Clinical and Economic Review (ICER)
Elektra Papadopoulos, MD, MPH US Food and Drug Administration (FDA), Center for Drug Evaluation and Research (CDER), Office of New Drugs
Eleanor Perfetto, PhD, MS National Health Council, Strategic Initiatives; University of Maryland, School of Pharmacy, Dept. of Pharmaceutical Health Services Research (PHSR)
Dana Gelb Safran, ScD BCBS Massachusetts (BCBSMA), Enterprise Analytics Ian Saldanha, MBBS, MPH, PhD Brown University School of Public Health
Divya Shridharmurthy, MPH University of Maryland, School of Pharmacy, Dept. of Pharmaceutical Health Services Research (PHSR)
Lee Simon, MD, FACP, FACR Harvard Medical School, Beth Israel Deaconess Medical Center, Rheumatology; Outcomes Measures in Rheumatology (OMERACT)
Jasvinder (Jas) Singh, MD, MPH University of Alabama, Birmingham (UAB) School of Medicine, Epidemiology
Vibeke Strand, MD Adjunct Clinical Professor Division of Immunology/Rheumatology, Stanford University School of Medicine Biopharmaceutical Consultant
Peter Tugwell, MD, MSc Department of Medicine, University of Ottawa, Outcomes Measures in Rheumatology (OMERACT)
Jacqueline (Jackie) Vanderpuye-Orgle, MSc, PhD Amgen, Inc., Innovation Policy Research Group, Global Health Economics
Lara Maxwell, PhD University of Ottawa, The Centre for Global Health
Rebecca Williams, PharmD, MPH National Institutes of Health (NIH), Office of Extramural Research, ClinicalTirials.gov
Paula Williamson, PhD University of Liverpool, Clinical Trials Research Centre (CTRC); Core Outcomes Measures in Effectiveness Trials (COMET) Initiative
APPENDIX C
LOW BACK CORE OUTCOME DOMAINS
ICHOM NIH ICF Deyo et al. Chiarotto et al.
IMMPACT (Chronic Pain)
Bombardier et al.
(Spinal Disorders)
Core Outcome Domains
Pain Medical historyBody structuresPain symptomsPain intensity Pain Pain
Quality of Life Physical functioning
Body functions Function
Physical functioning Physical functioning Back specific function
Work status Depression
Activities and participation Well-being Health-related quality of life Emotional functioning Patient satisfaction
Disability Physical examination Environmental factors Disability Mortality Participant rating of global improvement and satisfaction with treatment Work disability
Analgesic use Diagnostic testing Disability (social role) Participant disposition Generic health status
Adverse outcomes of treatment Sleep disturbance Symptoms and adverse events Satisfaction with care
Catastrophizing
APPENDIX D
TERMINOLOGY
The following definitions are adapted from the Core Outcome Measures in Effectiveness Trials (COMET) Initiative.19 We encourage project participants to familiarize themselves with the terminology and to maintain these distinctions throughout project discussions.
- Core outcome set – An agreed standardized collection of outcomes (representing one or more domains of a person’s health and function) which should be measured and reported, as a minimum, in all trials for a specific clinical area or of a specific condition20
- Domain – A domain is a broad category, an element of a person’s overall health and life in relation to their treatment (for example, mobility/physical functioning)
- Outcomes – Each domain contains one or more outcomes, which are the “what” we want to measure. Within a mobility/physical functioning domain, an example of an outcome might be joint impairment.
- Measurements or Instruments – Each outcome can be measured using measurements or instruments – the “how” something is measured. A measurement/instrument is a tool to capture an outcome. The tool can be a single question, a questionnaire with a composite score, a score obtained through physical examination, laboratory measurement, or observation of an image.21 In our example, there are several measurements or instruments to assess joint impairment e.g. X-Ray Petterssen score, change in range of motion, or Hemophilia Joint Health Score (HJHS) physician assessment.
- Measures – Scales or units used to describe outcomes. For example, range of motion can be reported in units of degrees, where normal range of motion varies from 0 degrees when the leg is straight, to 135 of bending. Alternately, the HJHS Total Score is reported as a numerical result, on a scale from 0 to 124, where higher scores are worse.
19 http://www.comet-initiative.org/about/overview 20 Williamson , PR. "Developing core outcome sets for clinical trials: issues to consider." Trials 2012, 13:132. 21 http://www.comet-initiative.org/glossary/cos/