By Adem Lewis / in , , /

(people chatting distantly) – Good morning, everyone. I’m Mark McClellan, I’m the Director of the Duke-Margolis
Center for Health Policy. I’d like to welcome all of you
to this morning’s workshop, or really, today’s workshop on a Framework for Regulatory Use of Real-World Evidence. This is a workshop that we are convening under a cooperative agreement the FDA and we’re very pleased to see
all of you here in the room and all of you who are joining us online for these very important discussions from a range of perspectives, government, academia, industry, research, patient groups and
others, for a productive exchange about this new and
emerging area of greater interest in developing evidence to improve the safety and effectiveness
of medical treatments that are available for patients. As you all know, there’s
been growing interest for some time now in
using so-called real-world evidence to inform key
clinical and regulatory questions about the healthcare system. This is due to I think
a number of factors. One is the increasing availability of data from the real-world clinical
practice and from patients. And also the availability of improving techniques for analyzing those data. Patients, providers,
payers are increasingly asking for more information
about the effectiveness, about the value of new
and existing treatments. And this evidence is
increasingly being influenced by what we learn in real-world settings. At the same time, regulators
and medical product developers are interested in identifying
what can be learned from these new sources of
information to help bolster the evidence around
particular medical products. And improve the efficiency
of the clinical trial process and potentially support regulatory reviews and a range of regulatory applications. The development and use
of real-world data though, is not something that’s new. This has been going on for decades. The FDA has used
real-world data in settings where traditional
randomized clinical trials are not feasible or not ethical. Think about applications
involving very rare populations for orphan
or ultra-orphan drugs or applications involving
drugs where there is an urgent unmet medical need and where good information on the medical history, the course of care for
patients may be available from real-world evidence is one example. As another example, in 2007,
some of you may remember the FDA Amendments Act
established a post-market drug safety surveillance initiative built around real-world data. That initiative, the Sentinel Initiative, now grown into an into an integral part of FDA’s drug safety
surveillance evidence work, is built on real-world evidence as well. So the application of real-world data and real-world evidence
within regulatory context has occurred, but it’s
met with some limitations. Limitations that you’ll hear about today that range from lack of
clarity about definitions and common understandings. And also, the need to
better characterize data that come from real-world
settings so they can be confidently used, or
more confidently used, in real-world evidence setting. As you’ll hear today,
there’s some very important data challenges and methods challenges in applying real-world evidence and developing real-world evidence in a broader range of
regulatory decisions. But this in an area where
there is a lot of interest and moving forward and why
this meeting is so timely. This is again, a reflection of the changes in data and methods technologies. It’s also a reflection of bipartisan interest, as you’ll hear. This in as an area where
FDA’s been called upon to take more steps in collaboration with the private sector to
develop better evidence, to improve medical treatments and to apply in regulatory uses. And it’s an area where,
because of the changes in technology, we have an
opportunity to move forward. So just to go over what we’re
going to hear about today, we’re going to start out in a few moments with our colleagues from
the FDA, who will give an overview of the agency’s perspectives on enhanced use of real-world evidence and progress implementing the
real-world evidence provisions in the 21st Century Cures
Act and the recently enacted Prescription Drug User Fee
Act, the sixth edition. We’ll then hear from my
colleague, Greg Daniel, who will provide an
overview of the current real-world data and
real-world evidence landscape. Particularly as it applies
to these regulatory contexts. And highlight the
framework that we provided in our recently published
white paper on the regulatory acceptability
and potential uses of real-world evidence in regulatory uses. I hope you all had a
chance to see that paper. It’s available for a
download on our website and on the webpage associated
with today’s event. We’re then going to be turning to four major topics that
are part of this effort to make progress on real-world evidence. That starts with a session
on real-world data. Data is not the same thing as evidence, as the white paper makes clear. And we’re going to
discuss some of the issues and limitations and challenges
that must be addressed in turning real-world
data into fit-for-purpose real-world data for these kinds
of regulatory applications. We’ll then have a session
on matching real-world data and real-world evidence into
regulatory applications, or use cases, real-world data and methods, that are fit-for-purpose for
important regulatory questions. Then a session on pursuing
real-world evidence development programs that
can support regulatory use. And this is where I think
a lot of the practical challenges will also arise. And again, in turning data
into reliable evidence using well-understood
fit-for-purpose methods. Finally, we’re going to close today with a session focusing
on the path forward to allow for broader
synthesis of the ideas and issues that came
up in today’s workshop. And a chance for some of our panelists and hopefully with
contributions from all of you, to discuss potential next steps to move this important area of work forward. Now there are a lot of
people here in the room, joining us online, collaborating, not just from Duke, not just from FDA, not just our working group partners, who are focused on real-world
evidence development. And we’re going to try to give you a good sense of those perspectives today. The lineup of speakers
that we’ve assembled have made significant
contributions to the background work that went into
getting us to this point. And significant
contributions to the agenda, the scope of topics,
that we’ll be discussing. So we’re hoping for a very rich discussion in pushing this challenging,
but very important set of real-world evidence issues forward. Before we get started,
I just want to mention a couple of housekeeping notes. As you’ll note in the agenda, each session will begin
by some brief remarks followed by panel discussion. We also have time set aside
for broader discussion, with all of you who are here in the room. For those in attendance,
we have roving mics for use throughout the day. We also to hope to hear from those of you who are joining us online. For those of you in the room
who don’t know it already, this is a public meeting,
everything that you say is going to be part of the public record to try to move these issues forward. And particularly for those
of you who are joining us on the web, we encourage
you to participate in today’s discussion in a couple of ways. If you do have any potential
questions for a panel member or for a discussion in
one of these sessions, please send them to [email protected] [email protected] and our staff will work to get them to us to incorporate in the discussion. And if you’d also like to join
the discussion on Twitter, and that applies to all of
you in the room, I think, too, please us @dukemargolis,
the @dukemargolis handle, and the #RWE, so #RWE. For those of you who are in the room, we have coffee and beverages
located just outside. Lunch is going to be on your own, we’re going to have an
hour break just after noon. There are plenty of restaurants close by and finally, I want to give you a reminder that this meeting is being
convened under a cooperative agreement with the FDA, but this is not a federal advisory committee. There are not going to be any votes. This meeting is going to be a success if there is an open exchange of a lot of ideas on these important topics. To get us started, I’d
like to turn this over to one of the leaders
from FDA on this topic, Rich Moscicki is the
Deputy Center Director for Science Operations at the Center for Drug Evaluation and
Research at the US FDA. And he’s going to start out
with some opening comments. Rich, thanks for joining us. – Thank you, Mark. Always a great pleasure. And I want to start out
by thanking, in fact, the Duke-Margolis Center
for Health Policy. Especially Morgan Romine
and Gregory Daniel for bringing this meeting together today. I have had the pleasure of following Mark in several of these
Duke-Margolis convening meetings. And it usually means that my
talk is reduced to, “Hello.” Since Mark usually covers
everything I was going to cover. So knowing that repetition is
important in adult learning, I’ll let you hear some of it again. I also want to take the
opportunity to welcome our panelists, the
audience and those of you who are watching on the webcast. This is maybe the sixth, seventh, eighth, meeting I’ve personally
attended on real-world evidence over the past few years. So what’s different? Well, what’s different
is that this is a very important step in the process for FDA. As we move forward with the commitments that we now have towards
real-world evidence, and its use in our own
context, this becomes a critical point in that discussion. Where we move from the
usual public discourse into the actual process
of getting real traction on using this within the agency itself. So we know that we and all of you, particularly sponsors,
share a very common goal. We want to make sure that
we get important medications to the American public
that are safe, effective, but in a timely and a
cost-effective manner. Now we realize the limitations
that our current process has brought upon us, using
traditional clinical trials. And so therefore, there
is a benefit to using data that’s collected
as part of routine care, in the patient’s own
daily life to inform us about efficacy of these treatments in what we are terming the real-world. So maybe real-world is a touchy phrase, because perhaps it
implies that the evidence that we’ve been working
with is somehow not real. But in fact, we all know
that this traditional method has brought to us
evidence of whether or not products do really work or not. And in fact, those products have benefited millions of patients. But we also know that the limitations are several fold for the
traditional process, right? We know that, for example,
only a small number of patients with any given disease usually can participate in such clinical trials. In fact, one survey said
that 16% out of 1,000 people who were interviewed had either
participated or knew of some family member who had
participated in a clinical trial. And that’s probably,
according to other sources, an overestimation of how many people can really participate
in such clinical studies. For example, even in cancer,
only 5% of cancer patients today participate in clinical studies, even when they’re in desperate situations. So this does raise some
meaningful questions about the representation
of the patient populations in randomized, controlled trials. Even though they do provide that clear picture of their effectiveness. Time and cost have been
a very big subject. They have risen tremendously over the last 20 years for clinical trials. In today’s era of we discuss the prices of prescription
medications, that heightens our sensibility around this
issue of the time and cost. So it’s important that we
continue to look for alternatives, at least in part to
supplement this process. Most importantly, though,
we must look for the innovative approaches that
allow us to ask more questions. With high-quality data to
answer those questions. And to do so in a very rapid way. So we recognize at FDA,
the importance of expanding the tools available to us for
getting this understanding of the risk and benefit of medications for the sake of the American public. Now Mark told us this wasn’t entirely new. And I think that’s very true. I’ll just reiterate that, in fact, we use real-world evidence today on a routine basis to
make regulatory decisions. Over the past few years,
we’ve worked closely with Harvard Pilgrim as
the coordinating center to increase our capacity
to use real-world data to generate evidence on safety
through the Sentinel System. And that relies predominantly
on claims and pharmacy data. We have incorporated that into the routine safety evaluations that
we now conduct at FDA. But we’re on the verge of moving further. The HITECH Act of 2009,
part of the American Recovery and Reinvestment
Act, launched the widespread use of
electronic health records. So now, by 2015, almost 100% of hospitals and close to 90% of office based physician have electronic records. So that begins to tell us
that we could fill the gaps that claims and other data
that we have had access to may not be able to provide us. But electronic health
records are not built with research in mind,
they’re for daily care. So not everything is ideal in our ability to just mine such data. So we need to adopt this technology for that research purposes. And that may not be straightforward. How the data is entered, stored, defined and terms are not completely standardized. To use such data for analytics
presents real challenges. So how do we similarly make
use of also mobile technologies that now access data on patients
that’s not just generated within the healthcare
encounter, but actually in the very daily lives of patients, as another aspect that we need to consider as we debate and move forward with the use of such real-world
evidence or data (chuckles) before we make it evidence. Also, such evidence then, if
it’s going to be seriously considered, requires verifiability. And a minimum of bias and
needs to be well-controlled. And we need to have
predetermined analytics. This is the basis of
our rules of evidence. But as many are aware,
the randomized trial has been the favored approach
to try to do all of this. So how can we act within this context to minimize bias in what
has been essentially by many considered to
be observational data. I hope we’ll get to the discussions today that we will look at
alternatives statistical and other methods to limit that bias that may not involve randomization alone. Now as the leadership of
FDA stated in a New England Journal of Medicine article last year, we can also use such real-world data together with a randomized trial. But there is no contradiction
between the generation of real-world evidence and randomization. And there are many other
ways that real-world data can contribute to randomized
controlled trials. Now also, I think you heard from Mark and I’ll repeat it
again, not every product that we approve has
been based on randomized controlled trial data alone. FDA has made use of data from registries, case series and expanded access
in its approval of products. Admittedly, most often
and ultra-rare disorders, where the sheer numbers
themselves make it difficult to conduct a traditional
randomized controlled trial. So here we are, this is a process. We have some new building
blocks and tools, but we still need to
understand how to build quality evidence needed
to support the findings of safety and efficacy that will continue to serve the needs of patients. Our continued dialogue, such
I hope we’ll have today, the collaboration that we
will require from all of us, and demonstration projects together with scientific rigor will be needed as we move forward. Today, I look forward to
hearing from the many experts that we’re going to have speak with us. And we will continue
to seek their guidance, your guidance, to construct that framework so that we may move forward
with the use of real-world evidence in our regulatory
decision-making. Thank you very much,
thanks for being here. (audience clapping) – Thanks, Rich, and as
usual, he said a lot of stuff that I didn’t say to frame
this meeting very effectively. We’re now moving into our first session, so after this framing,
we’re going to build up the framework for the regulatory
use of real-world evidence. We have two presentations for that. First is a presentation from
Jacqueline Corrigan-Curay, who is the director of the
Office of Medical Policy at CDER, at FDA, who is going to provide a bit more detail on FDA perspectives and framing for this work. And then Greg Daniel, from
the Duke-Margolis Center is going to give a presentation
on the real-world evidence landscape, key terminology, considerations for developing real-world
evidence for regulatory purposes, drawing on that background paper that hopefully many of you all have seen. So Jacqueline, looking forward to hearing from you, thank you. – Can I have the clicker? Thank you and good morning and I want to thank the
Duke-Margolis Health Policy Center for just pulling together
this fantastic meeting. And all our panelists who are here today to share their expertise
and thank you to everyone in the room and those on the web. So let me see, oh, it works, okay. So I’m going to just move forward. Quick disclosures, there’s
nothing to disclose. I’m going to talk a little
bit about definitions, some goals and expectations. I’ll summarize briefly our
experience with real-world data and real-world evidence
that Rich mentioned. Some foundational
activities that are already under way and how do we look
forward to where we want to go. So oops, I think I went
forward several slides. So the definitions, we
defined real-world data as data related to patient health status and/or the delivery of
the health care routinely collected from a variety of sources. And of course, that could
be electronic heath records and claims and billings,
but also other sources in home use or mobile technologies. And real-world evidence
is the clinical evidence regarding the usage and
potential benefits or risk of medical product derived
from analysis of RWD. So obviously, this is a regulatory definition of real-world evidence. And if you had the opportunity to read the recent guidance on real-world evidence from the Center for Devices
and Radiologic Health, you’ll see these definitions, so we have harmonized
on those definitions. And so the definitions
really provide a framework for the dialogue, but I
always think it’s important to articulate our goals
because otherwise we may become too focused on whether
this is pure RWE or not and lose sight of the reasons we care. And I think this was mentioned
before, but we really want to try and maximize the opportunities to have our regulatory
decisions incorporate data and evidence from settings that more closely reflect clinical practice. And that will increase
the generalizability, increase the diversity of population and hopefully also improve
some of the efficiencies. So we talked about,
there was some reference to some expectations, that
there are some expectations for FDA under 21st Century Cures. We’re to establish a program
to evaluate how we can use this evidence for the
approval of a new indication or to satisfy post-approval
study requirements. And the definition of real-world evidence in 21st Century Cures is
data regarding the usage or the potential benefits
or risk of a drug derived from sources other
than traditional trials. And you can see we’ve incorporated this into our definitions,
we’ve merely separated data for evidence so that
we can have the conversation about the unique aspects of those. We also have PDUFA
commitments to enhance the use of real-world evidence in
regulatory decision-making through conducting a public
workshop, such as this, to gather input on the topics. Initiate appropriate
activities, pilot studies or other, oops, this is going forward. To address key issues and to publish draft guidance on how real-world evidence can contribute to the assessment of safety and effectiveness in
regulatory submissions. So as it was referenced,
we do have considerable experience with real-world evidence. A lot of it is through our
work with the Sentinel System, which has really become FDA’s
national electric system, to look at safety data. And this slide just speaks to the data that’s available to us
through our data partners. And I think although we want
to think about efficacy today, we mention safety because
it’s really not only about looking at safety, it’s about making informed decisions and
generating evidence. So this is just one example,
when we started to get case reports of bleeding after
Dabigatran was introduced into market, the question was,
is it behaving differently now in clinical practice than
it did in the clinical trial? So using Mini-Sentinel,
which I’ll just refer to as a precursor of our
Sentinel, we were able to see, no, the bleeding rates
were similar to warfarin, just as they were in the trial. And it’s that evidence
generation that I think, that knowledge that we
can now leverage further as we move forward and look at using these data for evidence generation. And indeed, we’ve already
published some guidance that incorporates some of our thoughts on methodology and quality
and looking at data. But of course, we are here
to talk about efficacy. And I think it was referenced
that the leadership have published on this topic. And I think they recognize
certainly the value of this, potential value
for real-world evidence in the regulatory setting, but it also has to be thoughtful about
how we adapt these tools and methods of traditional trials. It’s not really abandoning them, it’s bringing them forward
into the new settings and being very critical
to make sure we obtain valid results and minimize bias. And that means a lot
of emphasis on what are the methods used and
what are the best methods that have been developed
and validated that can be combined in the appropriate
research settings? So when thinking about
how you turn RWD into RWE, there’s a number of pieces
that have to fit together. Is the data that you’re
going to be use, fit-for-use? What’s the quality of it? Does it even capture what
you need it to capture? As we said, electronic health
records are full of data, can you get that data
out of those records? Are there particular
regulatory considerations you need to think about
as you’re moving out of traditional clinical trial sites and decentralize and moving into practice. Data standards, how is this data captured and going to be transmitted? And then, what’s the study
design that’s going to pull this all together provide
the evidence that you need? I think we’re thinking
about all these factors and how it can be done
and I’m sure you are, too. And we would just urge, as
you start to think about turning real-world data
into real-world evidence, you engage with FDA early on this journey. So we have already started
on a number of activities to help us think through
the best way to do this. And I’m just going to, so, stakeholder engagement,
meetings such as this with Duke, are really great for us to
inform our thinking about this and we know that Duke
published the white paper that also will help inform our thinking. We also support the Clinical
Trials Transformation Initiative, and they
recently issued guidance, issued guidance, sorry, recommendations on registries and how to
assess those registries or design them for use
of real-world evidence. They are also engaging
in a project on mobile clinical trials, so how do
we use mobile technology to bring the clinical trial out
of the traditional settings? As I said, we’re working
on data standards, CDER and FDA recognize that we
need standardized study data terminologies to facilitate
efficacy analysis. That’s already under way and will help as you’re starting to
submit these data to us. We’ve also published a number of guidances that I would consider the
foundational guidances that tell you how to
move from a paper-based clinical trial to using
these electronic datas in compliance with our regulations. And finally, I’m going to highlight some demonstration projects
that are already under way. We talked about EHRs
and this really probably why we’re sitting here today. If there was not widespread adoptation, we wouldn’t have the data available to us. Or potentially all the data. This is a project that we’re supporting through a grant to Duke’s
Clinical Research Institute. It’s a HARMONY-Outcomes ancillary study. It’s an ancillary study to
the GlaxoSmithKline study, which is a cardiovascular
outcomes study with Albiglutide. And what Duke is doing is trying to assess at the clinical site, could the EHR, what was its ability to
facilitate recruitment, to populate baseline characteristics, so there are efficiencies, or even to identify the clinical endpoints. And I would refer you to the grand rounds, and this is available on the web, I believe this is the
new website from the NIH Collaboratory, to find out a
little bit more information. But this is the type of
information that will really inform us on how you use these data
and what they can tell us. I think many of you are
aware that there has been incredible innovation in oncology. And also, within the FDA’s
Oncology Center of Excellence, there’s a lot of very innovate work going on in the real-world evidence. This is a presentation at the ASCO meeting and reflects a collaboration
with Flatiron Health to examine how real-world data can be used to gain insights into the
safety and effectiveness of new cancer therapies. Dr. Abernethy is on the first panel and perhaps can speak more to this. And in June of this year,
another collaboration was announced between the
America Society of Clinical Oncology’s big data initiative. And all of this is being done through the Oncology Center for Excellence information exchange
and data transformation. And they’re really looking
at how we can incorporate real-world data and real-world
evidence into our decisions. Dr. Sean Khozin is the
lead on this project. The other thing, when you
think of medical records, you can think of going in
and extracting what’s there or you can think of trying
to start at the beginning and put the right data in so that we can use it for many uses. This is a project that we’re supporting, Dr. Laura Esserman, which is one source. Sort of enter this data
once and use it many times. It’s the integration of
standards based tools into the EHR to bring together
health care and research. So it’s another important innovative way to, sorry, this is just moving on me. To try and look at how we
can use the EHR for research and the demonstration is in
breast cancer clinical trials. And perhaps Dr. Esserman will
provide us more information. We talked about data
standards and this is, besides the data standards that
we’re developing internally, for submission and efficacy,
we’ve also seen the growth of a number of networks
that have real-world data available for analysis and
we’re thinking about how we can expand and leverage those networks. So not all of them speak
the someone language, the common data models,
but there are consensus based standards that
perhaps could be leveraged. So FDA is leading a project
called Harmonization of Common Data Models
for Evidence Generation. And that’s with NIH’s
National Center for Advancing Translational Science, the
National Cancer Institute, the National Library of
Medicine, and the Office of the National Coordinator
for Health Information. So the ultimate vision for this project would be to have all of
these networks be able to talk to each other and
provide data for questions. And it’s really to further leverage what we have already developed. And finally, I’d just like
to talk about evidence generation and we are supporting, now, a trial within the
Sentinel, so this is a first trial in the Sentinel System. It’s a practice and
patient level educational intervention to increase
anticoagulation use for individuals with atrial fibrillation who are at high risk of stroke. And some of the statistics
on the right hand show this is an important
public health question, but importantly, also for us, is going to be a pilot project that will
inform future interventional studies and really help us understand how these data can be
used to generate evidence. So I’d just like to say,
we understand that there’s just a wide range of ways
that you can incorporate real-world data into your studies. And it can be starting from the left and including improving efficiencies, to starting to integrate
it so it contributes to a greater extent to the
evidence all the way up to the observational studies in which there’s no intervention being done and it’s just extraction of that data. And I think for us, we need
to have continued dialogue to understand how this
evidence is generated and what it can inform in
the regulatory setting. So looking forward, I
think we want continued engagement with stakeholders
to identify the key questions that FDA needs to answer
to facilitate sponsor use of real-world data and real-world evidence for regulatory decisions
and that will help us provide appropriate guidance in this area. We want to continue to
identify knowledge gaps and support appropriate
demonstration projects, which we have been doing, to facilitate the development of RWE
for regulatory decisions. And develop a framework and program that is of use to everyone. I’d like to just acknowledge
some of the folks who contributed to these slides,
or whose slides I borrowed. And then I’d like to
close with just saying if you have questions and comments, we’ve set up a, I’m really sorry here, a mailbox for you to please submit your questions and comments to us. And so thank you. (audience clapping) – Thank you, Jackie. Let me, well, hold on. Okay, let me echo Mark’s welcome to all of you this morning. It’s great to have you all with us. I’m told that we have over 1,500 people on our web, so thanks to all of you for joining our session today. Over the next few minutes, I’m going to, let me introduce myself. I’m Greg Daniel, Deputy
Director for Policy in the Duke-Margolis
Center for Health Policy. Over the next few
minutes, I’m going to take a little bit of time reviewing the need for real-world evidence
to fill some of the gaps that we talked about,
as well as to present to you our framework that we’ve developed and presented in the white paper. As many of you know,
traditional randomized controlled trials have
long been the gold standard for evidence generation
related to the safety, and maybe if I talk a little bit slower, related to the safety (laughs) and efficacy of medical products. And randomized controlled trials will continue to be the gold standard. This is not about replacing randomized controlled trials, this is
about how real-world evidence can help inform regulatory
decisions and can help bring additional data and evidence to bear as decisions are being made. Yes, okay, however, these studies, the randomized controlled trials are, as we all know, increasingly
resource-intensive and time-intensive and often don’t provide a wide array of evidence
that might be needed for patient decision-making,
provider decision-making, and even payer decision-making. For example, evidence
on longer term outcomes or evidence on outcomes
that might be more relevant to patient populations
may not be developed during the clinical trial process. In addition, the population’s
represented in clinical trials may not necessarily reflect
the patient populations using the medical products,
including multiple comorbidities, different
age groups and other socioeconomic status indicators in groups of patients that may not
have access to trials or may not be well-represented. In addition, there are
gaps in our knowledge when a drug or device is approved for use. And increasingly
sophisticated data methods for filling those evidence gaps, through the use of routinely collected health data can be useful. And that’s real-world data
and real-world evidence. So right now, the nation’s
growing electronic health information
infrastructure has enabled routine and increasingly robust collection of such data and evidence. This data and evidence are often used for informing patient and
provider clinical decisions. And at the same time, payers and providers are moving more towards
payments and reimbursement models that are increasingly
reflective of value and linking payment value
and real-world evidence development can support
those decisions as well. Regulators are, as we’ve mentioned, are increasingly poised to make better use of real-world evidence and
the data and methods as the data and methods mature through
other uses and applications. As Rich and Jacqueline
have already touched on, Congress has mandated
FDA to explore the use of real-world evidence within
the regulatory framework, both in the recently passed
PDUFA VI commitments, as well as the 21st Century Cures Act. So two separate pieces of legislation governing this so FDA
really needs to do this. So when assessing the
real-world evidence approaches for regulatory use, it’s
important to make sure that we’re all using the same definitions and a common understanding of what we mean when we say real-world data
and real-world evidence. These definitions are
completely overlapping with the definitions that FDA put forward. And just to reiterate, real-world data is any data relating to
patient health status and/or the delivery of
health care that’s routinely collected from a variety of sources. This can include traditional
sources of real-world data that many of us think of,
like electronic health records and claims data, but also
it can include registries and data from mHealth
technologies and wearable apps. Or wearable devices,
apps on your smartphone, fitness trackers, as well
environmental exposures. It’s almost like every day, there are new potential sources for real-world data. Are they ready for research purposes? Probably not, and a lot of methods and curation do need to go into that. But the sources of real-world
data are increasing. Then the definition
for real-world evidence is essentially using research methods to develop evidence
from those data sources. Now real-world evidence can
be developed on anything. Medical products, new treatment patterns, new programs to improve quality of care. But as Jacqueline highlighted,
for regulatory applications specific to medical products,
real-world evidence can be defined as clinical
evidence regarding the use and potential benefits or
risks of a medical product derived from analysis of real-world data. Real-world data is not
simply anecdotes based on, or real-world evidence
is not anecdotes based on real-world data, but does
take, as I mentioned, more data curation,
standardization and methods to ensure that data we’re
using are fit-for-purpose or fit for regulatory purpose. As was highlighted earlier today, there have been, this is not new for FDA. They’ve been using real-world
evidence all along. In this figure, we try to
highlight some of that. Over on the far left side,
which is representing new drug approvals, FDA has occasionally used real-world evidence
for small populations. In the approval of a
new drug on the market, in a rare population where
it might be challenging to randomize the use of
historical or external controls, have been used in those situations. On the far right of that slide, we’ve heard about the Sentinel System. And the Sentinel System is routinely used to help inform FDA’s
decisions about safety of medical products and that’s entirely using real-world data sources
and observational methods. So what we’re really talking about today is what’s in the middle. So all of the kinds of
regulatory decisions that happen after a
product is on the market. New indications, labeling changes, new information to be added to the label, all of these require
a regulatory decision. But they all require,
there’s sort of a question, how can you use real-world evidence to support those decisions? The middle is where we are focusing and we’ve put forth a
framework in our paper that tries to at least start
the conversation about, when you ask the question,
can FDA use real-world evidence for what’s in the middle there? The best answer is, it depends. So what does it depend on? And what are the kinds of considerations that you need to walk through
to ensure that the types of real-world evidence that
you’re putting together are appropriate for the decision at hand. This is a framework that we’ve put in our white paper and it really does mirror well the considerations
that Jacqueline outlined. But essentially on the far left side, you have the major two considerations. Number one, what’s the
regulatory decision? Is it a new product on the market? Or is it a new indication
or a labeling change? And how close is the
approved product label that already exists, how close is that to the actual regulatory question at hand? If it’s the same population,
the same endpoint, but you’re adding additional
safety information to the label, or is it a
completely new population? It’s a little bit further
from the existing indication. All of those need to be considered. Then, what is the clinical context? Or area that the real-world
data will be collected? Certain disease areas give
rise to better data collection or more frequent encounters
with patients in the healthcare setting, so give rise to
potentially more complete data and a better ability to leverage
real-world data sources. Other clinical conditions
or clinical questions might have many more challenges
in terms of selection biases that may not be
well-addressed with the available observational methods
or other types of issues with the clinical context
that might give rise to different questions
about how real-world evidence might be supported there. That moves over then to the real data and methods considerations. Given the regulatory question at hand and the clinical scenario
that we’re talking about, can you actually collect real-world data that’s valid and reliable
for that purpose? Do we have the right quality checking and data curation methods? And then finally, from a
methodologic perspective itself, this gives rise to a lot of
different potential applications of methods to produce real-world evidence. Real-world evidence can
include randomized studies. Randomized pragmatic
trials that are conducted in the clinical setting, but still include randomization as well as
non-interventional studies or observational studies
that rely on existing data and populations that are not randomized. There are a range of robust and
reliable analytic techniques and study designs that can be used across these types of designs
and the question of, given the regulatory context
and the clinical context and the availability of real-world data, what are the best designs
and methods to apply to the data to get valid
and reliable estimates that would be useful for
regulatory decisions. These interlocking
considerations will guide whether or not real-world
evidence approach is adequate and appropriate for the
intended regulatory application. And again, we’re putting
this framework forward as a discussion tool to help
get today’s discussion going. We’d love to hear throughout the sessions, how applicable this is or
how you’re viewing this. We’ve built this through a planning group, an advisory group that includes folks from the manufacturer perspective,
the patient perspective, the methodologic and academic perspective and we would like to get a lot of your feedback today on this, thank you. (audience clapping) – Greg, thank you for
framing today’s discussion and thanks to Jacqueline, as well. You heard the issues that we’re going to try to cover during the course of the day. You’ve seen a couple of
frameworks that are useful for thinking about all these issues. And now we’re going to go
through these in more detail. I’m very pleased to be
joined on this panel, focusing on a development
of fit-for-purpose real-world data with
some experts in the area, from a wide range of perspectives. As you heard, real-world data is not the same thing as real-world evidence. But you can’t get to real-world evidence without access to and
understanding of real-world data that are valid and reliable
for the intended purpose. For this session, we’re
going to talk about some of the best practices
and key considerations for developing fit-for-purpose
real-world data. And I think we’ll also
hear from the panelists about some of the
challenges in getting from where are now to where we’d like to be with fit-for-purpose real-world data. We’re going to start out with some brief opening comments from our panelists and then have a bit of followup discussion and open it up to all of you as well. Let me start by introducing
this distinguished panel. At the far end is Kevin Haynes, a clinical epidemiologist on translational research for affordability
and quality at HealthCore. So HealthCore is a group
affiliated with Anthem that does extensive work in
real-world evidence development. Amy Abernethy, is the
Chief Medical Officer, Chief Scientific Office
and Senior Vice President for Oncology at Flatiron Health. There and in here previous career at Duke, very much involved in
developing richer data for reliable real-world
evidence applications. Sally Okun, is Vice
President for Advocacy, Policy and Patient
Safety at PatientsLikeMe and has led some of the
leading, or has helped guide some of the leading
efforts around incorporating patient data as a source
of, patient-generated data, as a source of real-world evidence. And Laura Esserman, who you
heard referenced earlier by Jacqueline as well, is
the Director of the UCSF Carol Franc Buck Breast Care Center and Professor of Surgery
and Radiology as UCSF and is involved in implementing
systems, as you heard, across a range of electronic health record and other data source networks. I’m going to turn to
Kevin to start us off. – Thank you, Mark, and thank you for the opportunity to speak today. I think we’re all interested
in real-world data to close these gaps in data
that it’s necessary to close the gaps in evidence to
ultimately to close gaps in care. So that’s one of the hallmarks of what real-world evidence can do. I’m part of the Sentinel
System, I’m one of the Data Core Co-Leads in
one of the pilot projects that you already heard about this morning with the IMPACT-AF project,
is a vital interest to showcase how we can
begin to close these caps in care through these
interventional studies. So these small pilot studies
that don’t feel small, but can begin to close those gaps. I think that some of the
key pieces of challenges that we’ll talk about today is
the fragmentation of the data and the fact that data is often held close to the institutions that create that data. And because of its both perceived value as well as the piece about business risk, with regards to data sharing. We talked a little bit this
morning about data standards and the things that need to
be done to close those gaps and how we think about
protecting patient privacy, the integrity of the use
and protecting the business interest of those data sources are three hallmark challenges that we, as we embark on closing
these gaps of data, will need to maintain. I take a philosophical view on this and take an extreme observational question to showcase the need to integrate data. So if you were to ask the question, what effect does antibiotics
have on the first two years of life and prostate cancer
or colorectal cancer. What data source, what
real-world data source would you use to close
that gap in evidence today? Well, it doesn’t really exist. My antibiotics are in a
closed pediatrician’s office somewhere in northern New Jersey. My child, Luke, can’t
even study this question unless all of his data still integrated from PEDSnet and PCORnet
all he way to Medicare data at the other end of the spectrum. So currently, I’m hoping
that my grandchildren will be in a position to observationally analyze that level of real-world data. I purpose two forms of data linkage. There’s longitudinal data linkage. People hit 65, they enter Medicare, there’s a huge disruption in the longitudinality of observational data. And the second data linkage challenge is in defined periods of time, the information that’s held
deep inside a clinical system, like vancomycin one
gram, Q12 on five south, that might wife might be entering
into an information system today at the hospital near the
University of Pennsylvania, is totally unavailable to a longitudinal picture of administration claims. There’s two major tenets
of data linkage from a gaps perspective that I see
in real-world data to close. And I’ll close with HealthCore
is very committed into this space, we’re
actually part of PCORnet, as one of the two funded
along with Humana, health plan research networks. And we’re actually also contributing to another pilot project with regards to the ADAPTABLE study,
which I think you’ll hear about a little bit more this afternoon, as health plans begin
to engage in the space of enrolling or recruiting patients into large pragmatic clinical trials. And I think we’ll hear about some of those themes as we go through the day. – [Mark] Thanks, Kevin. Amy, we’ll turn to you. – Fantastic, as Mark said,
my name is Amy Abernethy. My comments as it relates
to real-world data and solving this problem comes
from both my observations at Duke, where we were
looking at how do organize electronic health record
data within the context of a single health system as
well as at Flatiron Health, where we’re doing this both because we own an electronic medical
record as well as trying to aggregate data from
across medical records. Building on Kevin’s
comments, a few observations. One is that the EHR as
a data source provides the opportunity for a longitudinal view. But it’s often just pieces of that story. In order to create the story, you need to aggregate
the pieces across time. If the person is receiving
care in a primary care setting and then goes to the oncologist clinic, if they’re in different health systems or in different clinical spaces, you need to bring those
two pieces together. The second is the criticality
of getting data cleaned up. It is boring, messy work,
but it has to be done. There’s no magic bullet for the messiness of electronic health record data. And importantly, many of
the critical data points live in unstructured documents. So being able to process
PDFs and pull out, for example, information such as histology or biomarker status, is something
that we have to solve for if we’re going to have data
that serves our research needs. The third is linkage and data linkage is, again, a tough beast. We’ve shown at Flatiron that
we can link, for example, genomics data and electronic
health record data. But as you start putting
in more and more data sets, risk of re-identification
goes up dramatically. And we have to solve the privacy issues as well as the linkage issues. And the other think is
linkage is expensive. Another that’s important to solve for is data and context. Essentially, one of the
ways that we’ve learned, both at Duke and at Flatiron
to make sense of these data, is to generate patient
stories that represent what happens to people across time. Because individual
patient stories aggregate into cohorts or into populations. Being able to put data into context, you can understand when
the biomarker was done and how does that lead
to the choice in therapy and the choice in outcomes. That then takes me to my next point, which is real-world data is nothing if we don’t include endpoints. Mortality, in cancer,
we care about whether or not the tumor responded or came back. Patient reported outcomes,
as you’ll hear from Sally. So endpoints have got to be embedded in our data set, and again,
not for the faint-hearted. And then, finally, the
importance of having data that we can trust, where we’ve documented quality and reliability, and we also have provenance and the ability
to trace back to source. Because if we’re going to
generate these data sets, to then ultimately submit to the FDA, they’re going to want
traceability to be able to ensure that the information is indeed representative of what the clinician said or what we knew happened at point of care. So that’s another key
feature of these data sets. With that, I’ll turn it over to Sally. – Thank you so much. Thank you, Amy. And thanks for having us here today. I think what you’re going to start to see is sort of a theme. A theme of gaps, a theme of
needing to fill those gaps and finding different ways
of being able to do that. I’m going to put this in the context of the quote that you see up there. The process of developing
fit-for-purpose real-world data is really quite well-suited to this. If you’re crossing the
river by feeling the stones, it requires you to look
ahead to where you want to be on the other side of that river. It requires that we
incrementally and strategically navigate that river bottom
to find those right stones. The right supports that will get us across to the other side. This process is rarely linear. So we may in fact have
to go off course at times in order to be able to
find the right path. We might even have to
do something with others to get us across, find other
tools to make it across. Some river crossings
are going to require us to really sort out the
ways of collaborating and coordinating what we
need to do to get our plan to get to where we need to go. Defined broadly,
fit-for-purpose really means something that’s good enough to do the job it was intended to do or designed to do. The interesting conundrum
about that statement, in our discussion about real-world data, is that most real-world
data are good enough for the job they were designed to do. But they’re not necessarily designed for regulatory decision-making. Do we need to develop new real-world data that actually has been
specifically designed for this purpose of
regulatory decision-making? Or might we instead preserve the integrity and the original intention and
richness of real-world data and focus instead on
developing fit-for-repurpose? Understanding that that
will also require new tools, new methods and new understanding. What does it mean for a person
and patient-generated data? Certainly, a very rich
and increasingly available source of real-world data,
yet has been really not fully harness to its potential
to be fit-for-purpose or for that matter, fit-for-repurpose. So I’d like to have us
consider one use case. It’s fatigue in chronic illness. And more specifically,
fatigue in multiple sclerosis. Fatigue is present in about 80 to 90% of people who are living with MS. Fatigue is an individualized experience, yet it can be measured. Fatigue negatively impacts
a person’s daily life and can interfere with a person’s role and responsibilities
in ways that are very, very challenging for people to
live with on a regular basis. Yet, fatigue is rarely
measured in clinical encounters unless the patient themselves
brings that experience to the attention of the
clinician during the visit. Even then, it may not be
captured in the clinical record and the clinical code
might not be assigned to it for billing purposes and therefore it’s not going to be found in claims data. If a patient is prescribed
one of the two most frequently used products
for fatigue in MS, modafinil and, I’m not even going to try, you know, amantadine,
the connection to fatigue will not likely be found in prescribing or claims data because the
purpose for prescribing a specific product is
usually not captured. In fact, neither of these product labels include fatigue in MS as an indication. A recent systematic
review of these products used for fatigue in MS
found only 11 studies with sufficient effectiveness
data to be evaluated. The sample sizes ranged
between only 19 patients to as much as 121 patients. The review found that modafinil was not considered the more effective product. Yet a quick look at the
data within PatientsLikeMe, in our MS community with
its nearly 60,000 members, suggests that modafinil
is actually more effective in managing fatigue in MS. We have over 3,000 members with experience with these drugs who have
completed over 1,200 treatment evaluations for use in
specifically managing fatigue. These evaluations provide
data on start and stop dates for the drug, including
reasons for stopping, when that patient did stop the
drug and told us about that. Dosing and frequency,
perceived effectiveness for managing the fatigue,
the side effects, even though we’re not
necessarily talking about safety, we have some information
about the side effects people are experiencing with these off-label use and the severity. The overall burden of
using these products, the adherence to them,
the cost out-of-pocket, and a host of information in
narrative on advice and tips. Developing fit-for-purpose real-world patient-generated data
can begin by understanding the data sources that already exist. And evaluating its
capacity to be repurposed for regulatory decision-making. Fatigue represents a commonly experienced and often debilitating symptom across many chronic health conditions. My example is just one condition in which it can and is being
measured and is sufficiently important to assess as
an efficacy endpoint. In addition to the
traditional outcome measures in a range of chronic conditions. These data are not likely to stand alone. The journey to the other side of the river will require additional
tools and collaboration with other data holders to compliment and in some cases help
them complete their journey toward fit-for-purpose
of the real-world data that they have in their databases. So it’s time to harness the potential of developing fit-for-repurposing
patient-generated real-world data for use in
regulatory decision-making. The challenge now is
feeling for and finding the best stones and tools to get that data across to the other side of the river. Thanks so much. – [Mark] Thanks, Sally, Laura? – Thanks very much for including me, Mark. Our group is really trying
to refocus clinical practice on high-quality data collection
so that we can transform the point of care into a
patient-centric data hub where learning and
improvement are actually a part of the routine of care. As a surgeon, I can’t help but
think of a surgical analogy, which is first, stop the bleeding. I think that what we’ve heard is that all these problems with data really stem from basically the way
we practice medicine. We’ve been practicing the
same way for 70 years. Maybe it’s time for a change. If you harness a couple
of the ideas of quality at the source, enter once, use many, and you put this into medicine and really start thinking about taking back medicine to enable the clinicians to do what they went into medicine to
do in the first place. So if you actually,
again, not that you know every piece of information, and you don’t, and you can always go
back and try and get it. But actually, in a lot of conditions, we know the basic building blocks. We know the key pieces of information. And they can be assembled
into these simple checklists. That’s what we call the
OneSource Checklists. And it’s not just clinician
data, it’s patient data. Really, I don’t think
we should ask clinicians when a patient went into menopause. Actually, we should ask the patient. There’s a lot of things we ask that should come directly from patients. So the idea is if you
actually have that data and you can use it in real time, you would enable a host of
downstream improvements. You would enable quality
improvement, cost transparency, trade-offs, decision
support, trial matching. An ability for data to
seamlessly flow into trials. And I think that another
key principle is this idea of integrating research and care. We always talk about,
oh, well, there’s care and then there’s research. Really why are we practicing medicine? We should be practicing to improve. But you can’t improve if you don’t know what your outcomes are and you don’t know what things cost and you don’t know what patients opinions are about it. No business would ever run that way. But unfortunately, medicine is largely in the reimbursement
business, I’m sorry to say. And I think we just need to get it back so we need different systems. In terms of safety, I
think we have to have a lot more discipline
around adverse events. You can find it. In running the I-SPY trial,
one of the things that we don’t have is we don’t have
the adverse events built in. A lot of the thing that
are reported to the FDA are reported in the severe events of adverse events reports. But they’re not really in the record. How ridiculous is that? Why aren’t we tracking that? And another part of the problem
with electronic records is they were built to mimic the
way paper records were built. That means everybody
creates their own record. There’s no notion of a shared record. Amongst clinicians, let alone
clinicians and patients. And if you have an adverse event, you don’t need 25 different
versions of that event. You need one and you need some discipline. So that means, actually that we can’t just fix these problems from the back end. Someone, somewhere, we
have to go to the front end and say it’s time to change practice. That means mindsets, skillsets and tool sets have to be changed. And another big problem is
that the electronic health records are really institution-centered. And not patient-centered and
that’s another big problem. And it will bring up
problems with governance, which I do not know how to solve. But it is actually a
really important problem. We are trying to work across the UC system to makes this happen. To make this concept a reality. I think if we can get a
big organization to adopt this approach, and it’s
not just breast cancer, it’s just a good example,
but you could do it in breast, prostate, ortho,
doesn’t matter, orthopedics. It shouldn’t matter, right? And it’s going to take clinician leadership and patient leadership. And I’d like to say just
another two quick points. And that is the way in which
we conduct trials today can also help us merge fields and change. We’re conducting two big trials, one is a PCORI-funded trial to look at how to personalize screening. And we are actually designing
that trial from the get go to be institution independent and build in the adverse events and all
of the feedback in real time with payers at the table and
guideline makers at the table with the idea that when the data comes in, it will change practice in real time. And getting payers and
people to adopt up front, I would not say it’s an easy journey but we will get there. Another opportunity, I think,
is running the I-SPY trial, which is looking at people
of the highest risk cancers and trying to think about
can we use early endpoints to identify new drugs that work? Well, we’re at this critical juncture now, where some of our treatments have improved the chance of a complete
response three-fold. I ask you, why should I give
anybody the standard of care? Adriamycin, which patients call Red Death, would not graduate from I-SPY. Why should I be giving them that drug? So I think actually when
you have been running a platform trial for
years, seven years now, and I have really tight data
on what the standard of care is and I have biomarkers
that allow us categorize the tumor types and know
what their outcomes are, why do I need to keep
assigning the standard of care? Why can’t trials and
platform trials emerge and evolve into real-world
evidence generators. And if you take your early endpoints, part of the reason why the early endpoints haven’t worked is people don’t keep them, people who have these endpoints,
all the way to the end. And that’s actually how you prove it. Your early accelerated approval
would allow you to say, okay, I’ve got a good response. And that long-term follow up actually should be your final approval. Using a base of historic data
that is tightly designed. I think it’s an interesting
opportunity to think about how we evolve the trial system. Because Lord knows, the
confirmatory phase III trials with eight to 10,000
patients is not working. It’s a waste of money,
it’s not helping us, we have to change. I think we have to
broaden our ideas of what is real-world evidence,
but it has to start by transforming the clinical care process. We should not be practicing
with the SOAP note anymore. It’s time to move on. – Yeah, well, what you’ve laid out is not only about
transforming the clinical trial process, but transforming care so that both are based on–
– Correct! And by the way, who needs data most? The clinicians who are taking care of you! – That’s right, that’s right. And I’d say for all four
of these presentations, it’s nice to see both that the passion and the practical side, so
you can see a vision here of how real-world evidence driven by, at its core, reliable,
timely access to needed data. Use once, or sorry, collected
once, used many times, could really transform clinical trials, evidence development and
especially patient care. As you’ve heard, we’re not there, yet. And I want to go back to some of the, a little bit more discussion of some of the key reasons that you raised. I’d like to start with
some of the challenges, just around the complexity and reliability or lack thereof of the data itself. And then turn to some of the institutional or maybe business interests. Kevin, I think you said proprietary issues that may be in play here. And how particularly this additional FDA effort around regulatory uses of real-world evidence can
help transform all that. As well as the data can help the regulatory evidence succeed. So let’s start with some
of the data challenges that the data itself
that you all mentioned. I heard over and over again
that healthcare data is complex and messy and hard to
collect consistently. And that goes to some of the
comments that you all made about free text entries,
electronic data that’s based on an electronic version of a paper record. I know from working with
Rich Platt and Sentinel, there are, I think, Rich,
something like 100 ways to report on a blood sugar hemoglobin AIC test, and some
of what you all described was, I don’t think maybe brute
force is not the right word, but just hammering these
data into standards through some combination
of developing algorithms that can take all of
these different formats and turn it into a data element that’s cleaned and more reliable. Laura, you mentioned developing lists that could expand that
over time, checklists, that start with maybe a limited
amount of key data elements but grow, and structured.
– That are structured. – And grow from there. So that may be one path for it. I’d like to hear people thoughts about how to address that. I’d also like some thoughts
about some of the ideas that you see from other
Silicon Valley colleagues who are very interested
or talking a lot about about data lakes and
freeform text and using additional tools to take those messy data and nonetheless, turn it
into something that’s like, like you said, Amy, kind
of a story for a patient. Even though we don’t have standardization on the underlying elements,
at least we can get to effectively, perhaps,
effectively standardization on the stuff, the results that matters. That might be, I don’t
know if that’s another really feasible approach. I don’t know if these
visions for data lakes are really souped-up data
swamps, but I’d appreciate some further thoughts
on how we can actually, we’ll get to the business case and the governance and
things like that in a minute. But just, how we can go faster. It seems like a pretty daunting task given the complexity of healthcare data. – A couple of comments here. This is where we need to
learn from the tech industry or actually have the tech industry help us solve this problem. A couple of key points,
I like the language you used about hammering into standards. The truth is, what we need
to do is systematically go through and clean things up. When you’re talking about Rich’s example with the many definitions of glucose, getting to single
concepts that have single sets of units that we can analyze. One of the things that
we’ve learned at Flatiron is that as you start
to develop the process, you reuse that process many times. So you learn how to get
albumin into a single concept, you can now reuse that for glucose, you can use it now for another
set of biomarkers, et cetera. So one is, that process of
developing or hammering out the standards can be
used and reused itself, if you take some of the
core principles of building algorithms, technology,
and putting that together. But the second part that you brought up, and you were talking about
the issue of essentially aggregating through data lakes, or swamps, there is no magic there. You still have to have a system to which you’re going to make sense of it. But once you do make sense of it, one of the things I can
show you is how quickly you can then start drawing
insights from the data. One of the ways that we
focused on making sense of it, within the context of
Flatiron as well as at Duke, has been by using patient stories. Because if you can think about
a person with lung cancer and know in general this is
how it looks when the person gets diagnosed at stage II,
when they have their surgery and radiation, when the disease
ultimately comes back and you do a set of biomarkers
that then dictate chemotherapy. And then potentially
hospitalization or death, you can now start to generate
a set of data stories that allow you to link data
together in meaningful context that readies for analysis. The last two pieces I want to hit on, one is that we have a belief system that data need to be perfect. And in fact, they don’t
– They don’t. – We’ve been working
with claims for 25 years and we know how to work through claims. We have same kinds of things
we need to work through with electronic health record data. We can do simulations to
test, when do you need really highly reliable
data and when can you get away with less reliable data? But more importantly, one of the things that we’ve learned over
and over again is the more that you work with data and analyze it, the cleaner the overall
system of data gets. Because the two pieces reinforce. That’s my last point,
which is that ultimately, the analyses, you can
just wait for the data to be perfect before you do the analysis. Because you do the
analysis, you compare it to what you would have expected to get, for many core analyses, and then you use that information to update the overall general set of data. – I’d like to maybe make a comment. You think about the
transcontinental railroad. This is where we are now. And these are the efforts
that we have to make to try and make things work. We should take those learnings and say, honestly, how do we
want to see the practice of medicine in the future? Truly, we’re not going to
be able to screen scrape our way into a different
future, partly because you’ve got to change the
mindset of the clinician. By saying, okay, you don’t have to change anything you do, just keep
doing what you’re doing and we’ll just fix it all on the back end. That’s of course what
you have to do today. But really what you want
people to think about is oh, what are, do I need albumin? Do I need hemoglobin A1C? Okay, just build it into
what I’m thinking about, but get people receptive and ready to say, oh, how many people did I
take to the OR last month? How many times did I take
them back for re-excisions? Getting people to start–
– Is that where your checklist comes from, you start with these clinical questions?
– Absolutely. – And then get them to
help you generate the reliable data elements?
– Yes, we have been able to generate the checklists
and people can agree on what the key data elements are. In fact, when we brought epidemiologists, all the scientists together and
all the clinicians together, even regardless of specialty,
you want to know what? Everyone needs the same data. It’s actually not some great surprise. The trick is trying to turn
them into structured lists and actually figure out
a framework where people can actually use it in clinical practice. And the reason it’s so hard is means you have to totally re-engineer the way you run your practice and think about, it’s actually quite, it is
feasible, we’re doing it. And actually, I have to say
that the current way electronic records work is the reason
why we’ll be able to fix it. Because people are suffering with the burden of documentation. And I actually have that to
thank for enabling change because I think the relief you’ll get, plus all the things that
you’ll be able to do with it. You gotta get people excited
about the opportunities and say, wow, what could I get back? What could I learn? I want to be a part of
the process of learning. We have to step it up
here on a clinician side and say that, yes, we can do this and yes, we can make a change so that
the future can be brighter. That we learn about all of
these problems and solve it. I would say, to me, the biggest problem is going to be governance. Because the problem is, where does the source of data lie? Who governs that? Right now, it’s been convenient that institutions do
it or companies do it. But people get their care, as you said, Kevin and Sally, across
all kinds of places. So where is that right central repository? Or how is it that everyone has to make their data open and searchable. These are really important
and difficult issues. – They are, I want to
get to, just to pick up on this point about are
there key data elements that in a particular,
keeping with the framework that we talked about earlier, a particular fit-for-purpose particular use case, that could be agreed upon. And maybe, Sally, you might go back to your example of multiple sclerosis. You’ve done a lot of
checklists in the context of breast cancer and other contexts, but for multiple sclerosis,
for generating real-world data, do you see a path that could include? You’re familiar with those patients, that could include some
of the key clinical data that Laura was describing, but also key patient-reported data like fatigue. – Absolutely, and for
those who are not aware, PatientsLikeMe is moving
into a whole new realm of data in terms of
thinking about integrating new sources of data through
our DigitalMe initiative. Which is actually collecting bio-specimens from our patients and MS
is one of the conditions within which we’re doing that. What we’ve done is we’re parametrized that entire experience of living with MS and we start to understand
what are the state changes that people go through over time? And how can we begin to
biologically understand what’s happening at that
point of that state change. And patients actually
putting their hand up and saying, “I had a flare and I want to “better understand what’s going
on,” or, “I had a relapse.” So the opportunity for us to suggest that the phenotypic, the
phenomics of the experience is something we’ve been
at now for 12 years and have been collecting
in a very codified way. In ways that, actually, the FDA has become much more aware and understanding of, given our research collaboration with them over the last few years that it’s not so different from what some of the data is that they’re already accustomed to, when it’s collected, curated
and coded appropriately for use in other settings
beyond what we have. I think the other piece
that we feel is really critical is that the
source here is the person. If there were going to be a governance, I’d say it’s the patient. I’d say it’s the person. And we have to find a way of being able to have someone, whether
it’s you and your son going in and trying to
determine an antibiotic use or myself going in now
and closing in on becoming a Medicare beneficiary,
to say I want to know that these are the things are important to me at this age my life. And here’s the things that have happened to me in the past, and how
have we learned from that? So I can have, going
to my older age feeling like I’m confident that
I’m going to access to the type of care, the interventions, and the total whole health that I would expect at this stage of my life. – I’d love a comment from you, too, Kevin. Go ahead on this, then I
want to push on the data linkage issues, so even
if you have the list of elements that you’d like
to put together, how we do it? – It requires others.
– Yeah, so to take and build on the fatigue example,
I think claims data is longitudinal, but delayed, right? So it does set itself up for
good, robust environments to do some of these safety
analyses that we’ve talked about. Clinical data is deep, but not complete. Because you don’t have the information that may have gone last
week across the street. I’m still on Vicodin because
it’s active med in my EMR. So I don’t want to get that transferred all over the place, because I’m not. And then patient data is complete, but it’s not standardized
and it’s not codified. I think all three, payers,
patients and clinicians have a stake and I think
that there’s going to be the demand on the change
that, if I as a payer, knew that the patient was in fatigue, how can I provide better
care for that patient? Maybe I send them a Lyft to help them get to their appointment. Maybe I provide an
additional services to them to add value to them. But I don’t know that unless the patient tells me that or is using some app that may only be at UCSF
that then I can’t see that data and provide action on. When we as a health system
close those gaps in data, then we as researchers
and evidence builders will have the available data and be able to draw inference and
evidence out of that. The day supply field in
pharmacy data used to be really bad because it wasn’t audited, it was where the pharmacist’s
hands hit the keyboard. Once it became audited, it
became really, really clean. And it helps Sentinel
immensely at being able to define episodes of care
and those types of things. So sometimes when you drive things into the different payment models, you’re going to improve
the data environment. If we improve the data environment, we will undoubtedly be able to improve the evidence that comes out
of that data environment. – So for sure, we pay people for doing 12 point review of systems. If you paid them for
completing a checklist, or for filling out the adverse events, things would change overnight. If you said, oh, I’ll
do accelerated approval. And I, the FDA, will
make it available if you fill out adverse event
report forms, that’s safety. And payers said, oh, and I’ll pay for it, only under coverage with
regulatory evidence. Meaning, you have to
fill out the outcomes. You would get the data you
need and things would change overnight, because people
do what they’re paid to do. – I see Kevin nodding, but Amy, your thoughts again?
– I’m more dubious (laughs). We studied this at length at Duke. One of the things that we asked was, if we build systems and force
people to put information into systems, how many data
points could you get people to structure on the front
end before things fall apart? It turns out that it is highly related to your seniority in the system. That much is for sure. But the answer was somewhere between three and seven data elements per visit. So you gotta decide exactly what three to seven data elements were. Why, because even if you fixed your screen and said you can’t enter
any more chemotherapy until you put in this
performance status data point, they’re just going to put in
anything to keep advancing the screen unless it meaningful
to their clinical practice. – Or it’s audited.
– Okay, but I agree with that. However, let’s think about it. That’s if you don’t change
the way people practice. For example, if you sent out a survey to get information from people. If that could start assembling the record. If your nurse navigators or whatever, instead of writing notes
and throwing them away or putting them somewhere,
actually the whole clinic, it’s going to require a change. I didn’t say everyone
doing the same old thing. If you can do this in way
that then when you come in, if you think about the the
30 people that are employed to touch a patient and
do all these things, if you actually change their job and said, the primary thing is getting
these data pieces correct and that would be how, that
we would be paid and fueled, and you start using from
the beginning to the end, then you could change. I didn’t say it was going to be easy, and I didn’t say, but
I think it’s possible. But you can’t ask people
to keep doing everything they’re doing plus fill
out all this stuff. And you can’t do it at the back end. It has to be in the process of care. It has to be instead of and you have to get something back for it. – So I submit to you, think
about this way, Laura. We’ve got a 10 to 20-year vision, where I am in 100% agreement. We have got to, really, as you said, re-engineer the whole thing. But we’ve got a 21st century care mandate that says we’ve got to get these things figured in the next two to three years. Let’s figure out, in the
next two to three years, how to get our data to
the point where we can get it ready for a series
of responsibilities that we have in front of us. And then let’s set the
vision of how we’re going to start to meet these things, so that we don’t have
to continuously hammer. I think that was the word (laughs). – Even when you think about
biomarker data or whatever, you think about it as raw
process and signature data. And you could almost think
about, you at PatientsLikeMe are collecting a bunch of
information that you’re trying to gain insights
and eventually it will distill into, oh, this becomes something that ought to be routinely captured. And it becomes a signature. These are all, there can be many sources. But you distill out not
every piece of data. Again, you’re not looking for perfection. You gotta like really focus on, again, each person collecting those maybe five to 10 elements, period. That’s it.
– That’s all I can do. – That’s all they’re
going to, but that’s okay. You can do that because you don’t ask any one person to do everything. – I can see a pathway where, I guess the business case, you
might call it, is changing. As Kevin mentioned payments
changed where actually getting better results for patients, or tracking how they’re
doing longitudinally matters. The potential business
cases around regulatory uses of real-world evidence,
as you mentioned, Laura, would be a potentially strong motivator. If there was clarity
about what kind of data elements needed to be collected. And maybe that would help
overcome the linkage issues, too. But there’s some other
reasons why we’re not seeing these data come
together, aside from linkages and consistent
sharing, reliable sharing, being technically difficult and at least burdensome in the current system. You all, Laura mentioned governance, how do you see that the different parties that need to come together for this, working more effectively together? Payers, patient data sources? Health system data sources? All seem to be key contributors to these critical checklist data
elements that would be the foundation for reliable
real-world evidence studies? Kevin, do you want to start? – Yeah, so governance is what I spend 80%, I think, of my day on. In trying to figure out how to conduct data linkage. Let’s be honest, the
health system already knows how to effectively
communicate real-world patient identifiable data to
the insurance company. Because UCSF or Duke likes to get paid. And Anthem and other insurers
like to know who they’re paying, so there’s already
a handshake that’s out there that’s really, really valid, it has to be. It amazes me looking at
the back end of claims now, having been a pharmacist at CVS going, these things are getting rejected. I’m amazed at how things
get into the data sometimes. But there’s already the pipes in place to do really robust linkage. I think it takes, from a
societal perspective then, being able to repurpose
those already robust linkages to then provide for a
framework to not only improve patient care, but then
also build on the back end. You talk about the five data elements that I might be able to
tease a provider with. If one of those is, did
they get their flu vaccine? That’s currently wasting
a lot of time in medicine. Because if I just got my
flu vaccine at my workplace insurance, that should
automatically turn off the alert at the UNC or Duke system. – Yeah, that’s right.
– But those little micro seconds of delay are killing us in time with the patient because
that data’s not integrated. Not the least of which,
it’d be great to have real time vaccine
exposure data, of course. From a governance
perspective, we absolutely have to come to the
table and figure out how we can recontract, reunderstand that the needs need to be
there for both the patients, but then also for the
research for the evidence to be built for a broader group. – I’d like to just touch
on a couple of things here. I think one of the things
related to governance is that currently, whether it’s in PCORnet or some other areas where
we’re actually gathering some patient data from
patients and thinking about how that works, a lot of
it’s being framed within the common data elements
that we’re familiar with with Sentinel and other
databases that becomes part of. There really is not a repository right now of patient-generated data within
which other kinds of other information could start to
be aggregated and collected. So I think that that’s
actually a real opportunity for us to begin thinking about. One of the things that
I think, and the way Amy describes it, is this
value of the narrative and the value of the data coming together to tell a complete story. But if we are only depending
on common data elements, we will not get a complete story. We have to find a way
of being able to pull in these other pieces of
information and ultimately, really nuance and add
richness to that context. In ways that other sources
of data cannot fill that gap. That has to come from
the person themselves or the caregiver themselves
to better understand what is that real
experience when you’re not in the view of the clinical encounter? When you’re not in the system? When you are actually
at home or in the place where you reside trying to
manage in your day-to-day life? How can we help people
gather kind of information and have that feed into
a repository of data that ultimately goes beyond
this common data model of standardized data, which
I think actually ends up losing the story.
– I think Laura and Amy may have some comments on that. – Yeah, so I’ll give you an
example from our WISDOM study. This is Women Informed to Screen Depending on Measures of Risk. And the interesting thing is we have built in patient-reported outcomes and that’s the easiest thing to get. Because we’ve done doing it going forward. The hardest thing, I
mean, the one data element that we we need which is res density, and if we are linked into
the hospital, we can get it. And for people who are coming
from outside hospitals, that one piece of data is killing us. And it’s not even like it’s hard to get and we’re going to do a
natural language pass, and can get it in, but the whole process is about how you link people everywhere. We’re going to get it from payers and then we would get it a year late. And you can’t get it in real time. Again, there’s this lack of ability to just have, okay, there are a few basic data elements that you
really need to have. And because you don’t have them, you don’t have them organized, and there’s no standard or requirement to have them exchanged or transferred. It’s really hard, all
this stuff is really hard. All of us would say that. It’s amazing we’re still
standing, but it is. – We persist.
– And that’s action, that the industry should
crumble because of that. They should know we’re out
here and we’re going to solve these problems, anyway. But I think that again, if
you just had some easy data, ideas for data exchange that
people required to do that, if you use IHE and force
a data exchange and say, okay, the reason why
I say it’s so critical on certain data elements that
should be easily exchangeable, no matter what system you’re
in, it would really help. – I agree for the core
need for a parsimonious data model and I also
have seen, in spades, the importance of being able to add in the extra level of context
and detail that can’t be gleamed from a common data model. But I want to reflect on the governance question for just a second. When I was at Duke and I’m
looking right now at Adrian, governance was like 90%
of our conversation. Because, frankly, Duke could not do UNC. – [Mark] Kevin said 80%
and I heard Adrian laugh. – My patient needed to
nine miles down the road, to be in a clinical
trial, and I’d always say it’s easier to get in the car with your medical record and drive it down than I can possibly figure
out how to exchange. The problem was really
because of alignment of incentives around governance. Now that I find we’re in a
very interesting perspective. So we aggregate data across our network, both on our electronic
health record as well as electronic health records
outside of our network. But then clean it up and
give back to the providers. Frankly, we have almost no
conversation around governance. We all agree that the data
really belongs to the patient and the providers, that
Flatiron’s responsible for getting it cleaned up then we license all the extra clean data
elements back to the sites. But at the end of the day,
because of the alignment of incentives, I’m a site that
wants to do quality measures and understand data and
be able to measure things within the context of my practice, there’s already incentive that has me within the governance model, in a way, that I didn’t have to
worry about when really it was Duke and UNC fighting
it out across state lines. Or really, in state lines.
– In state. – So the other part of this is that I’ve watched it get
progressively more complex as we started linking in data sets. So when, for example,
we link in genomics data and it’s highly aligned,
because we’re now thinking together how we’re going
to do research together that the organizations whose genomics data are being linked to our
electronic health record data, we actually have no governance issues. But when we start doing
linkages where there are people at business
odds with each other, the linkages get really tough. So it’s actually putting on the table, what are the business issues?
– Absolutely. – And talking through it, because often you can come to a business solution, but you have to be really honest about what that’s all about
before you get there. So if I’ve learned anything, making this jump into the tech industry, it’s been how to actually
manage the governance issue. – Yeah, this does seem like an area where there is some more
potential for convergence between these real-world
evidence data needs and the shift that’s
taking place in payment towards more accountability for results and the shift that’s taking
place in patient expectations towards more ability
to see quality of care and outcomes that really
matter to them, too. And again, it seems
like the regulatory uses of real-world evidence could help add on an analogous or an additional source of momentum for these changes. But I don’t want to leave
this issue of governance and increasingly rich data
linkages without talking about privacy concerns and patient
trust concerns as well. So somebody is holding
this increasingly rich set of increasingly accurate
data, which is potentially good for improving patient
care, potentially good for giving patients insights
about what they should do next, how they should make decisions. Who is holding that and
how do we pay due respect to patient concerns about
trust and use of that data? Especially as we extend from patient care, which is governed by one set of rules to research which is
governed by something else? – In this week’s news
alone, on any time you have these major breaches of
like Equifax, you know, you lose my banking, that’s okay. It really sucks, but I
can rebuild my financial. But you lose the fact that
I’m HIV positive or something, I can’t get that back. That’s out there and that’s permanent. So absolutely, patient
privacy is at the forefront of every one of these
discussions on governance. Because it is this most
sacred information that yet, patients all the time are
joining Facebook groups and chatting about, but
that’s their decision to share that information,
they own that decision. If we’re sharing data without the context of protecting that patient privacy, and it is lost or used in a way or manner that was not governed under the governance that needs to be put into play, you don’t get that back, you don’t
get that trust back. You lose that member,
that member complains to their employer, you lose
that business, frankly. So patient privacy is of the
absolute utmost importance in all that we do when we have
the governance conversation. – I want to be cautious, though. There is absolutely no
way we can assure people 100% protection of their privacy in the current environment that we live. I think we need to be honest
and transparent about that and remain open to the
ways that we can ensure that the people who are
the stewards of that data are held to really high standards. That we have very intolerable thresholds of breaches of anything that would suggest that you did not do the
absolute best you could to keep my data as safe and
as protected as possible. With the understanding that
you know what, stuff happens. And there are going to be
times when that kind of absolute protection is not possible. I think the public knows that. I think patients, especially those living with chronic illness and
especially those living with really acute and serious illness, are willing to, they
have a benefit-risk ratio that’s very different than those of us who are healthy and thinking
about how we protect our information in different ways. For their perspective, care
and research are intermingled. – Exactly.
– They don’t think about this distinction between,
oh, research means you have to generalize it, oh,
care means it’s for me. No, from their perspective,
we should be doing all of that together in
a way that ultimately gets us to the outcomes
that improve my health. – [Mark] That was Laura’s point, too. – That is my point
exactly, because in fact, I don’t think you should
be allowed to practice without an eye towards improvement. And I think you have
shown at PatientsLikeMe, that people are absolutely
willing to share their information, with some
risk of loss of privacy. – Big responsibility on our part. – Especially at the beginning,
to try and get people to pay attention to things
that they were important. So I do think one other
thing that is actually interesting is that a lot
of specialty societies have now been emerging as
the holders of the data. There’s this huge ophthalmology database. There’s one in cardiology,
– I sense some skepticism coming up (laughs). – Well, I’m just saying,
Cancer Link is now trying to organize all of the cancer data. I have to ask, is that
really the best place? Now I realize it’s not
the era of big government, but I have to say I think this is a role that the government should play. Whether it’s the FDA or it’s the, I mean, you answer this question, Mark. I think that there has– – I’m just the moderator. I get to ask the questions.
(audience laughing) – I’m asking it back to you. But it’s like there has to be
a place where we can sort of, where there’s some standards
for who’s going to hold it. The problem is that everyone’s
got these hidden incentives to do things, there has to
be a safe, trusted place for data to stay.
– Let me turn this one to Amy, since I’m the moderator. But it sounds like
– Oh no. – two things that are coming
out of this discussion, one is that patients are willing to share, but they need to have some confidence, that whatever system is
helping to bring this together and support their care
and support research is doing it responsibly and
effectively and as safely as possible given that
nothing’s going to perfect. And second, some way of
maybe actively saying, yeah, I want to participate in this study, that’s not maybe the
traditional way we’ve done that kind of informed consent. And I don’t know, Amy, if you
were going to comment on that, but this is my effort to turns questions back to the panelists (laughs). – A couple of comments. The first is, I think patients do often have a genuine interest in sharing, but they also have an expectation that the system has their
best interests at heart. If you read the Equifax articles, I don’t know how many of
us sucked in our breath as we got to the place in
the article where it says, and some of the senior
execs were selling stock before the event happened. I think all up until then,
we were like, oh, gosh, you know, this is sort of like a foreign government breach kind of thing. And then all of the sudden
we got to that point and we we’re like (snarls). And I think that patients expect of us to absolutely be coming at
this from the point of view of we are in it together to protect. The second thing, and it
goes back to your point, Laura, I don’t care if it’s the government or medical societies or anything else, whoever’s doing it,
there’s the expectation there’s a technical
solution to keep data safe. The ophthalmologists, it’s
not the ophthalmologists who are building the vault for the data. It’s actually really a technical company that’s working with them. So we need to be drawing on the best of technology to solve this problem, including new innovations for the future. There’s a meeting going
on not long from now, asking can blockchain help? We need to be really
thinking about all the different ways to solve
for creating the vault. The third part of this is
something that hasn’t come up, which is within the context
of our current privacy legislation, HIPAA, and
how that may get updated in the future, I’m not going to go into, but in the context of
that, it is really hard to de-identify data because
if you essentially just take out the named identifiers,
it doesn’t do the trick. If you’re going to do statistical
or third party review, the truth is there are
not enough third parties statistical reviewers
in the country anymore to actually be able to
verify all the data sets that are being generated. So we actually don’t have
a really good system, even just to manage the
requirements of HIPAA in a way that is keeping
step with the data sets that are starting to come out. So we need to make sure that we’re driving all these pieces forward
in a way that basically goes back to your core point. Getting to the place where we’re doing what patients expect of us. I imagine a day where it’s just like, on my Facebook, I’m able
to designate what I believe I want to have happen with my information. If somebody manages it
poorly, I can rescind that responsibility of the right
and in an electronic format, but we’re not quite there yet. – I think we have to be a
little safer than Facebook. – Yeah, well, maybe, (laughs) but you kind of get my point, that the toggles are clear to me. – It’s interesting, for our
partner for the WISDOM study has been Salesforce and one
of the reasons we picked it is because they actually
have such fabulous security. And the idea is we’ve tried
to create this as a registry that payers have access to,
patients have access to, the providers have access to. That it would serves
as an ongoing registry. But if we’re really successful, and again, billions of dollars are
spent on screening tests. One, alone, it’s like a $10 billion a year and aggregate business. If this becomes a national
registry, where should that sit? Should that be at the CDC? Could that be the FDA? Should be under PCORI? But it should be somewhere. If it’s successful and
grows into something that becomes this ongoing
learning registry, then it’s gotta transition from a trial to something that’s what? – It still needs the
right technical framework. – [Mark] Technical framework,
trust, effectively. – We’ve got a good technical framework, we have it, but again,
it’s going to grow up into these things.
– There’s nothing for it to sit within.
– Eventually, these things, if we’re successful at all, where does this evolve to? Where is the framework for this? – I get the sense I’m getting
asked a question again here. And I think I’ll actually
take this panel’s discussion as being a success in laying
out this important challenge and what the characteristics
of the solution look like. So we may not, in this panel,
during this hour today, get to exactly what these
long-term solutions are, but I think you guys have done a great job of teeing up not only
issues, but the path forward. And what the properties are that a solution needs to look like. Which should help advance
real-world evidence. Now we do have some time,
we could go on for a while, but we do have some time
for questions and comments from those of you who are here. A microphone is going around. Marc, do you want to start? And then I see a couple of
other hands up here, too? So, right here, first. With Steve. Steve, if you could just announce, just tell people who you are. – Hi, sure, Paul Bleicher from OptumLabs. And first of all, I have
to start with a disclaimer, because otherwise, you’re going to think that this question is a little out there. The disclaimer is that
I come from the world of regulated clinical trials and archiving of data at Humedica and running very clean,
precise studies at OptumLabs. So I get everything. But I’m playing off what I
heard a little bit of today. And it feels like we’re
at Yahoo versus Google. If you remember the old, old days, and we know who’s the winner there. Yahoo had this carefully archived and constructed categorization of the web and Google came in and said, “You don’t have to do all
that, just search it.” And find what you need and
have a good search algorithm. So we are in the middle of a revolution in data in the analysis of data. And in healthcare data, specifically, around the use of artificial intelligence, specifically deep learning. Which is fabulous at
natural language processing and doesn’t require you
to clean up the data or to categorize or
standardize it or whatever. Should we be skating to
where the puck is going? Or should we wait and be like the healthcare industry
and pharmaceutical industry have been traditionally, which is, let’s wait 20
years and we’ll adopt this. Should we be thinking
about a different way of analyzing data than classic
frequentist statistics, but more make use of this very new, but very powerful way, and other ways of looking at data that can potentially give us better answers,
not require us to spend all our time getting our
data into perfect alignment? And be able to operate more
efficiently and effectively? – Paul, thanks, and Amy, you
want to do a short answer? And we’re going to take
this, again, the purpose of today is not necessarily
to solve all these problems, but to get the challenges and a potential path forward laid out clearly. – So I think this is a critical question. It is where the puck is
going, in a lot of computing. However, within health,
we have the responsibility of getting it right and having systems that we know can get it right. If we think about artificial intelligence, we’ve got basically supervised
and unsupervised learning and the risks of unsupervised
learning just really don’t necessarily hold up right now. But if we can create data
sets that are appropriately labeled that we can use
to train our algorithms, and then build algorithms that start to essentially nibble away at
the edges of these problems, we can systematically get there. But it cannot be blacks not clearing. Healthcare requires of us to
show that we can get it right. – Great, thank you, and
great topic for the future. I got Joe and I think
Rich had a comment, too. – [Joe] Hi, Joe Selby from PCORI. Outstanding panel, lively
and right to the point. Kevin, I think it was you
that used the word societal, that we need a societal change. And Sally, it was you
who talked about patients and their true inclinations
and feelings and concerns and what it would take. And Laura and Amy, you talked about, I’m not going to remember exactly, I had it, it was all so nicely set up. But the question is, you all alluded to the notion that we need
a big societal change. And I have come to exactly
the same conclusion. That whether it’s IRBs or whether it’s institutions talking to each other. Would anybody care to just
name some of the steps along that societal change pathway? – I’d be happy to address that. I think the idea that
you have to go to an IRB to look at your data to learn and improve is offensive to me. That should be part of what you do as being a physician or a clinician. I think the notion of
the idea that we should have a system for research
and we should have a system for care, I
think is also outdated. So I think that we have
to start saying that, we have to start getting rid of, both in our public
speaking to patients, oh, research is something you do over there. The reason why nobody
participates in trials is because we’ve got this big divide and because we have
imposed this ridiculous, onerous structure on how we run trials. Which is with huge overhead. If we instead, okay, I hope next year I’ll be able to do things
better than I do it this year. So because of that, we’re going
to ask you some questions. We’ll make sure that you’re
data is available to you and you can get back to it
and we’re going to provide opportunities for you be in registries, for you to be in trials and
we’re going to make it simple. We have to start rolling
some of these ideas out. People have to start
embracing ways of saying there’s a new way of practicing. We have to start training
physicians differently. We have to give them different kinds of, a different mindset, as I said, mindsets, a different set of skillsets and a different set of tool sets so that people can do these things. And it’s not going to be perfect, but we can’t just keep, it’s important to clean
things on the back end, but it is important to change the frame. One of the reasons why
I’m trying to do this at the University of California is because they have six medical schools. And it’s one of the ways
in which you can start training the next generation of physicians to think differently and
do things differently. But I think it’s also all of us, in the way we speak about this. The way you speak about
what data is from patients and why it’s important
to get data from them, but also back to them. One of the things we’ve learned
from patients is they’re willing to give us data if
we give them something back. They want the information back themselves. So more of a shared experience. And that people should
be expecting to learn and look at different options and be able to have that information themselves. – Just as a–
– Can I just? One thing I’d like to add is that in fact, it’s not just a societal
change, it’s a social contract. – Yes, correct.
– That we do need to realign around and
from what we’ve learned from patients, it’s their
interest in understanding how they think about their health is how my mind and body are doing. And help me measure that
and help me understand what that looks like,
whatever the measure might be. And then also, we’re working really hard on this notion of thriving. And when we’ve asked
patients what does it mean for you to thrive in the
presence of an illness? What they’ve told us is it means
I can live the life I want. So if we can help people better
understand how to maintain the health that they have, even if it’s in the presence of illness,
you can still have health. But then put that into
some framework that allows them to put that in context to live the life they want.
– And what can they do to really get better?
– That’s realigning the social contract around
each of us, frankly, not just patients, but just
each one of us in this room. – Yeah, that is a model
of continuous improvement and how we support patient,
continuous improvement in knowledge and think it
will be very interesting to see as this real-world
evidence work moves forward, how we make the incremental steps. We’re not going to get there overnight,
– It’s required part of practice, patient data, – But what’s the path forward?
– clinician data, integration of data.
– I think we have, let’s see, was it Rich? Rich and okay, go ahead, Rich. And then Marc and then. Rich, you may not, I
may have to cut you off. – [Rich] I have my ways. Hi, Richard Platt, I lead
the Sentinel Initiative. And I agree with all of the above. But I would say 95% is the right figure for how much is governance
and how much is technical. (group laughing)
– It’s getting bigger. – [Rich] One of the key
contributors to Sentinel’s success has been that we basically
side-stepped that major issue of what data will you provide
and how will we secure it? Because Sentinel is founded
on the notion that FDA asks questions of the data holders and they answer the questions. But they almost never provide
individual level data. If they do, it’s in a
very circumscribed way. I have been astonished
over the past 10 years at the progress that our
methodologists have made in being able to do increasingly
sophisticated analyses without actually having
their hands on the data. Or putting the data into
a single pool data set. These distributed analyses
are getting closer and closer to the kinds of
analyses you do with pooled data. We now have a working
prototype of a distributed regression model that
should allow us to do any regression without putting
data into a single place. At the moment, it works
across Sentinel data partners. Can we do this kind of thing across Anthem and Humana and Etna? Both the theory and
the prototype also work for distributed analysis
where it’s linking the Anthem claims data
to the PEDSnet EHR data without any of those sites actually releasing their grip on the data. And it seems to me that
I’m in favor of everything you suggested, but we also
ought to look for ways to be able to work with
the data where it lies. – Yeah, in distributed models. I’m going to leave that
as an excellent comment to again help drive this forward. And maybe Marc and Rich, if you both make your comments or questions and then we’ll see if we have time for a response. – Marc Berger, this has been a fascinating conversation, but it seems to be that it’s been about, and I’m an IOM fan, this is about a learning
healthcare system. This is not about
regulatory decision-making. In a learning healthcare system, you can do what every big tech firm does. You can do quick AB testing. Or you can do control charts to see that if you make a decision, are you getting the outcome that you want to be getting? And why you get the right answer, it doesn’t matter as long as
patients are getting better. The right patient getting the right treatment at the right time. The FDA doesn’t look at it that way. They make a decision and they have to, they don’t want to make a wrong decision. So if they were to give
credence to some data, but it turns out to be
not reliable enough, then we’re going to
have regret and they’re going to get blamed for it. So they don’t have the
ability to go back and forth and make it go, so it’s
a clash of cultures. – But I don’t–
– Let me finish. – Yeah, we’re going to
finish and this again, this is not going to be the
last word on these issues. We were just hoping to get
the discussion moving forward. – I think it has to do with the fact that we think of a regulatory approval as a big cliff that happens in the life of a treatment that’s being developed. We have to move beyond that. And the system is going to move beyond it. Because the learning healthcare system, I know everybody wants
to wait until we have the perfectly curated data,
and like, it doesn’t matter. The data is good enough today, it’s being used by
everybody, a lot of people, a lot of institutions, today. And they will change the practice of medicine much more rapidly. The question is, can the FDA have some say where it looks I can now
let people know about how the practice of medicine has changed? And I can let people know
about it in the label? That’s the question? – And that gets back to our topic. And I think actually you
and Laura are largely in agreement on this notion
of continuous improvement via data and evidence.
– It enables everyone to have a different role. – So I want a last word
question, Rich, over to you. – [Rich] Well, maybe the
advantage of going last is that everybody’s already
touched on the questions I had. Particularly the last
two sets of questions. Who knew this is complex, right? (panelists laughing)
(audience laughing) And I think we heard that in
a very excellent discussion. And again, my congratulations
also to the panelists. We’re under a mandate and we have outlined the pathways. And I would ask the
panel to maybe close with what’s the timeline on such a pathway? And what are the quick
wins that will allow us to really move forward in a meaningful way in a shorter timeline? – Yeah, let me ask that as
a great closing comment. And I ask you each to keep
to 30 seconds or less, to keep us sort of on schedule. But a quick next step to get
us moving down this timeline? – Governance and getting
the legal pieces in place to make these linkages happen. It’s what I spend 80% of my time on. I try to spend some time
with my kids (laughs). – So while I agree with Laura
in the long-term vision, we can get data sets ready right now, including with the parameters that should be able to meet the expectations of OSI. So three quick wins, we
should be able to use these data sets for a label
expansion or a label revision. And what that happens
in the next 18 months, I think that will be a huge change. The second is when we put these data sets and serve them up as a
part of a pragmatic trial that has regulatory intent. And that will be another
substantial piece. The third is when we’re able to use these data sets to study populations and we’ve actually already got a
really good example of this, who otherwise could not have been studied in the clinical trials. For example, patients with heart failure who are in need of a breast cancer drug with potential heart risks. So I think those are my examples. – Great.
– Yeah, and I would follow up on the label expansion. I think we can start working
with that as a real potential target using patient-generated data. Especially that data
that’s being collected in a systematic,
methodological, codified way as a early use case, to
determine whether or not the data we’re gathering
has insights enough to then look at other data sources that, as a compliment, or some
other way of creating evidence from a host of different sources together. But not including patient-generated data, not trusting that it
actually does have value in a place within this evaluation,
would be a real big miss. And I think it’s an
opportunity not to miss. And I think we can do
that in the next 18 months without working together. – I think certainly once you
make changes to the systems, everyone can start to
have their roles evolve. And for the long term, I think the FDA could behave very differently if they had different systems they could rely on. But in the short term, I
think the platform trials are a fabulous opportunity
to say we can allow a therapy to evolve and we can think about accelerated approval with
regulatory evidence generation. And you could start taking,
allowing trials that have had really clean data
collection with early endpoints, evolve into dropping standards of care that actually
perform in an inferior way and allow people to continue to go on where then you’re just starting to continue to gather the evidence. You have your early endpoints, and you follow people to the end. I think that is an easy early win that would be a big game changer. And make people say, oh, I
can change the way clinical trials are run, period and get faster. – All right, I want to thank the panel. A long-term vision on data.
(crowd clapping) Practical stuff to get there. Thank you all very much. All right, we’re going to
take a 10 minute break, try to reconvene at about
11:15, thank you all very much. – [Woman] Hi, how are you? (people chatting distantly) – [Mark] We’re going to start
again in about two minutes. Two minutes, so those of you who are out in the hallway,
please start heading back. (people chatting distantly) All right, we’re going to start
up again in just a minute. I’d like to ask everyone
to take their seats. Just as a little bit of logistics, administrative information,
while people are headed back. This session is going to
run for about an hour. And then we’re going to break for lunch. Lunch is on your own, for
those of you who don’t know this area, around 17th
and F, there are lots of restaurants in the area
and we hope to see you back in a hour after we do our lunch break. Okay, so for this session, we do need to get started
to stay on schedule. I know there’s a lot more
that we need to discuss today. For this session, we are
focusing on matching real-world data and real-world evidence
to regulatory uses cases. And I think a big part of this discussion is going to focus on specific
types of real-world data. As we focused on in the last
session, but connecting to study designs, to analytic methods. For those of you who read the white paper, you saw there was just as much emphasis on methods being fit-for-purpose as the data themselves being fit-for-purpose. And that’s going to be the
focus of our session right now. So how can we match methods, study designs and tools for analyzing data
to the real-world evidence for use in specific regulatory decisions. So with that framing, I’d
to introduce our panelists. David Thompson is Senior Vice President for Real-World and Late Phase Research, at IMC Researcg, InVentiv Health. Marc Berger, next to him
is the Co-Chair of the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in
Healthcare Decision-Making. Which has done its own reports on related real-world evidence topics. Jeff Curtis, is the William
Koopman Endowed Professor in Rheumatology and
Immunology and the Director of the University of
Alabama-Birmingham Arthritis Clinical Intervention Program at UAB. And Adrian Hernandez, my colleague at Duke is Professor of Medicine and
Vice Dean for Clinical Research at the Duke Clinical Research Institute. As in the last panel,
we’re going to start out with some opening framing
comments and a few slides from our panelists and
then we’re going to go into discussion and again, I want
you all both in the room and online, if you email
us to get questions into the discussion
that’s going to follow. With that, I’d like to
turn this over to David. – Well, thank you, Mark. Oh, very good, and thank
you all for attending and it’s my pleasure to be
a part of this discussion. Very important event we’re having here. So as you could see
from my slide template, I’m a representative of
the Real-World Evidence Project on the Clinical Trials
Transformation Initiative. And so I did want to open
with a few words on that. It was alluded to earlier
today, but we didn’t really go into it too much. But as the name implies, the idea is that we’re trying to transform the way in which clinical trials are conducted. So the CTTI framework, it’s multi-stakeholder,
it’s evidence based and it is designed to have
measurable impacts on the way in which clinical trials are performed. In was founded about 10 years ago, co-founded by Duke and FDA
and has about 80 members. Now the Real-World Evidence Project team is listed here, the CTTI Project Manager, Garret, is actually in the audience. I put his face here
without his permission. (audience laughing) So that if you want to find out more about what we’re doing with the
Real-World Evidence Project, you can hunt him down
later and he has assured me that he did not exhaust
all the names at this list before he finally got to
the bottom and chose me to represent the entire group. In any event, so far what we’ve been doing is focusing on some of
the definitional issues with respect RWD and RWE. And a set of interviews
are being implemented across various stakeholders
to try to better understand their perspectives on data
needs, evidentiary requirements. Specifically to help decision-making and of course to incorporate
this into the way in which clinical trials are conducted. And there’s been some discussion earlier today already
about the inefficiencies and the high cost of clinical trials. With the data that we have available now and the technologies that are available, there’s got to be some
ways in which we can bring these together to develop
some greater efficiencies. So I’ve been asked specifically to comment on use cases for real-world
data in early development. And the interesting thing
about it is I’ve been doing retrospective database work
using real-world data sources for nearly 30 years now
and for the most part, it’s always concentrated on
products already on the market. Phase IV, because by definition, you can’t really collect real-world
data on a product until it is actually on the
market and being used. But there are instances
and we’ve some references to them today about the way
in which real-world data can be used for products
early in development. Specifically from the
regulatory standpoint, we’ve already talked about
how on an occasional basis, FDA will accept historical controls from real-world data sources. And this tends to be in instances in which it’s either infeasible and/or unethical to create a control group
for a particular trial. Rare diseases is the most
common example used there. Beyond that, my work has
focused on helping manufacturers understand the competitive
landscape in which they are going to enter
as relates to using real-world data to understand
things like treatment patterns and drug adherence for products
currently on the market that they may be competitors with as their clinical development progresses. And so the questions that arise relate to, is there effectiveness,
efficacy-effectiveness gap? Are there differences between how products perform in clinical trials versus how they perform in
actual clinical practice. And that’s a phenomenon that
has to be taken seriously. We need to measure that,
we need to assess that. And it can be used to inform the next wave of clinical trials for competing products that are coming down the pipeline. Also, with the size and vast volumes of data and the greater
numbers of patients that we’re seeing receive products, once
they’re out on the market, there’s greater potential for
doing patient segmentation. An assessment of heterogeneity
of treatment effects. So we might find that there
are particular patient populations that aren’t
benefiting from a product. Even though they might have
been represented in the trial population, but are
being washed out by the average treatment effects being
reported in the clinical trials. We have the capacity, we
have enough cell size, across a range of patient strata. In order to be able to
look at differential benefits and harms of different drugs when used in clinical practice. And we use that as a basis
for assessing whether or not there is additional
unmet medical need. In addition, there’s
been a lot of discussion and it seems like a low-hanging fruit area would be, for 21st Century
Cures, is to potentially use real-world data for label extensions. So what we’ve done historically,
though, essentially in the category of hypothesis
generating activities. There’s a whole body of
comparative effectiveness research in which we’re able to compare products using quasi-experimental
design techniques. And some of the other statistical advances that have come down the pike to address issues of confounding and bias. And we’re able to do
product A versus product B comparisons and assess
treatment outcomes and costs. Historically, that’s been
specifically for hypothesis generation activities
that could be confirmed in later clinical trials. But potentially, in the
future, this is an area where FDA might actually evaluate this kind of research for
decision-making purposes. And then finally, I come from
a CRO and we frequently use real-world data sources to
assess protocol feasibility. Specifically, this relates
to when a protocol comes down the pike and we have
inclusion/exclusion criteria. We can take those
criteria and overlay them onto an existing database
to see how stringent certain of the criteria
are, in terms of filtering away the available patient population. And we could use that to
tweak the potential criteria for selection of patients
so that we have a feasible protocol to implement in the field. Now when we start talking
about where things are going in the future,
there’s some reference to this in the white paper that’s been
cited a couple times today. And one of the things we’re talking about within the City Transformation
Initiative for clinical trials is to what extent,
not only the data, but the IT systems that house the data. To what extent can these
be used to transform the way in which we are
conducting clinical trials? So we can use these data
systems to identify patients who might be candidates
for inclusion in studies. And this really shifts the
trial paradigm on its head. Because historically, we
identify sites and investigators first and they enroll patients. But now we have the capacity,
using the technology, using the data, using
our analytic methods, to look for the patients first and then reach out to the physicians. We have also, through those
electronic medical records systems the communication
channels in place to then reach out to those providers. And the providers are the
only ones who the patients are because all of our analyses
are done on encrypted data. So the patient identifiers
aren’t available to us, but they are available to the providers. So it’s a two-step
process for that outreach. And then finally, once the data, once the study is under way, we establish some connectivity between
the EMRs themselves and the electronic case report forms to try to automate some
of the data capture and reduce redundancies in data entry. I have a little graphic
that summarizes that and I’ll just demonstrate that
once you have an EMR system, the communication channels can
go in the opposite direction. You liked that, huh, Mark? I always want to amuse Mark. (Mark laughing)
(audience laughing) And so you have the capacity
to use the provider network that is in place as a consequence
of them all being united in their use of the same
electronic medical records system. And then you use the
communication channels that are established
electronically between the EMR provider and those practices
and then have then, collected the patients
included, it’s everyone in the practice on the one hand and it’s selected based on protocol in another. Provider-induced variability
in data collection. Big differences, practice-based customization of data collection. It’s encouraged by EMR systems. It’s highly discouraged in trials. The project that is ongoing,
so those are my opening comments and thanks very much. – [Mark] Thanks very much, David. And next is Marc. – So I’m here today representing
the ISPOR task forces that have reporting, we’ll have two papers out this month online. And just to let you know what ISPOR is, in case you don’t know,
it was founded in ’95, with a mission to promote health economics and outcomes research excellence
to improve decision-making. And it has to grown to
become one of the leading global scientific and
educational organizations. And as we heard in the last talk, there are many different stakeholders. And ISPOR thinks that all the groups around the circle need
to part of the solution as we build a learning healthcare system. All right, so the challenge
of real-world evidence. There is tons and tons and gigabytes and gigabytes of real-world data. This is a picture of the background radiation in the universe. We think it’s reliable,
even though it wasn’t collected in a randomized
controlled clinical trial. But how do know when a data
is reliable and trustworthy? That’s the central question for the FDA. Is it good enough? Is it reliable enough that
enables you to make decisions beyond what you’ve been
able to make thus far on safety or for rare diseases? To make real-world evidence useful, it has to produced in a quality way, there has to be good data collection, good analytic methods,
transparent study procedures to enable replication,
good procedural practices, which I’ve called study hygiene. And then responsible consumption, which means informed interpretation and fit-for-purpose application. What I’m going to focus
on here is the last two sub bullets under quality production. Because I think there has been concerns regarding the replicability
of observational studies. And whether or not there
isn’t some data drudging that goes on to cherry-pick good results. So how do you know that hasn’t happened? As I mentioned, this month, coming out in joint publications in Value in Health and Pharmacoepidemiology and Drug Safety, there are two task force
reports for joint task forces between ISPOR and the
International Society of Pharmacoepidemiology, ISPE. One about recommendations around good procedural practices to ensure that you think that there is greater trustworthiness around results
that you get from a study. And one focusing on the replicability, so that you can say, hey,
if I had the same data set, would I get the same result? I’m not sure if it was a
year ago or two years ago at one of these Duke-Margolis meetings, I put forward the fact that we
had good procedural practices in place for RCTs and they were needed. We wanted to make sure
that pre-approval RCTs were pre-registered on a public
website, We wanted to make sure
there was completion of an a priori protocol
and data analysis plan. Transparent documentation for any changes in study procedures. And expectation that all RCT
results would be made public. There were no well-accepted
recommendations for good procedural practices
for observational studies or real-world data studies. And we saw this as a gap that,
it’s not the whole solution, but as one step, if they’re adopted, that would lead to greater
confidence and trustworthiness given to these kinds of studies. Now a few groups have
been talking about this, about pre-registration,
but it needs to address not just publication bias,
because we know journals like to publish positive
results, not negative results. And it has to address data dredging. Can you torture the data ’til it confesses and tells you something
you want it to say? And then there are other concerns around internal validity,
inaccurate recording of health events, opaque reporting. We believed that following
the RCT-like practices is a logical starting point. Now we’re not trying to
tell everyone that every observational data analysis that you do needs to follow these rules. So we divided the universe of RWD studies into two different categories. Exploratory studies and
hypothesis-evaluating treatment effectiveness
studies or HETE studies. Listen, everybody don’t,
when they start playing with data, they don’t always
know what the strength of the data is, what the
weaknesses of the data, what questions you can ask. You explore the data and
you generate hypotheses. Nothing wrong with that. But those are not the
studies we’re talking about. If, on the other hand, you
want to an observational study that you think should
inform decision-making by any healthcare
provider, then you should be having a a priori hypothesis and a established protocol
because you’re going to evaluate the presence or absence of a pre-specified treatment effect and/or its magnitude. And if you did that, and
you said you were going to do this in advance, then
I think that you should have greater confidence that those results are not the results of random findings, but actually represent as
good as we’re going to get. And if you don’t rely upon a single study, but say, let’s now see
that in multiple studies, you know what, that’s
kind of what a learning healthcare system’s going to be. So we’re going to focus
on the HETE studies. And here are the recommendations. That HETE studies should
be pre-registered. A study protocol analysis
plan should be on a public registration site prior to
conducting study analysis. The publishing of study
results with attestation to conformance and/or deviation from the original analysis plan. Everybody modifies their
analysis plan to some extent, even in clinical trials,
as they go through. You should be very explicit about you did, why you did it and what you found. There should be the opportunity
to replicate findings. Many of these data sets
are commercially available. So if you’re transparent enough
about how you did the study, other people should be able to go in and get the same result. If you do an exploratory
analysis off of a data set, you shouldn’t use that data set to then go do a confirmatory analysis. Why, because you may just be showing what you found in your
exploratory data set is not generalizable beyond that data set. So it should be done in a different population when feasible. Now sometimes it’s not feasible. If you do a study off of
the Sentinel data set, you could be hard-pressed
to find a data set as large and rich, Platt-ish, as that. Hi, Rich.
(people chuckling) I think the authors should work to address methodologic criticisms. This is an evolving area. The methodology is rapidly evolving and there’s still questions about is the methodology always good enough with a particular analysis? I think a robust public discussion about what are the pros and
cons, need to happen. And then including key stakeholders, not the least patients, but caregivers, clinicians, administrators,
HTA/payers, manufacturers. It should be relevant
information to everyone who wants to see a
learning healthcare system. The ISPE-led report focused on enhancing reporting guidelines
to identify the minimum set of items necessary to report in detail to achieve a fully reproducible evidence from large healthcare
database cohort studies. I think we’ve been
pushing in this direction for a while, but I think this report goes into much greater detail
than has gone into before. I’m not going to go into
great detail about this, but suffice it to say,
there’s specific decisions around analytic data extraction from raw longitudinal data with a
focus on temporal anchors. The minimum reporting of
independent investigators to be able to reproduce
the data cohort study. Starting from an analytic data extraction from a longitudinal raw data set. And the reporting on the analytic cohort before and after adjustment. There are many things in this report. It builds on some of the
work from RECORD and others and many good recommendations
that have come out from various groups around
the country and the world. So closing thoughts, I
think as a first step, enhancing the trustworthiness
of real-world evidence requires us to have good
operational procedures. To make us believe that
this wasn’t the result of data dredging and that
people know explicitly what you did and actually, they can go out and reproduce what you did. I think these are good basic steps. It will not take place without a variety of stakeholders,
including journal editors, regulatory authorities, providers, payers and HT authorities putting
the right incentives in place. But as we heard before, and I don’t think it’s only about money incentives. Yes, money talks, nobody walks. But there are many kinds of
incentives that could be good. The HITECH Act, look how
it’s changed a number of EHRs that are being used in this country in such a short period of time. With the right incentives, you
could move people’s behavior. We have an upcoming meeting
that’s going to be on October 20th, here in Washington
at the Grand Hyatt Hotel. Which will be jointly
hosted by ISPOR and ISPE. In which we’re going to
present in complete detail the findings of these
two task force reports. And begin to start a dialogue
with various stakeholders about how we can get the enterprise of real-world data
analysis to resemble more what has been done for clinical trials. Thank you very much. – [Mark] Thank you, Jeff? – Well, thank you very
much for the invitation to come spend some time
with you and I guess by way of disclosure, if not
apology, I’m a rheumatologist. So most of the use cases
I’m going to describe are out of my field that have
personal experience with. But the principles are generalizable. I was asked to take some of the concepts that we’ve speaking about
throughout the morning and bring a few of them to life. And sometimes when these topics
come up in conversations, they seem very futuristic
and it all sounds great, but who’s actually doing this and what are the regulatory implications
of doing this type of work? Do you have real uses cases or is this still all in a theoretical framework? So I’ve taken four different
domains as described on this outline to just
bring you a few examples. Pre-licensure or an
expansion of an indication, pragmatic clinical trials
and our two other topics. So I study mainly
rheumatoid arthritis or RA and psoriatic arthritis and conditions that are fairly uncommon that have a population prevalence of roughly 1%. So if you want to study safety
and comparative effectiveness in an uncommon disease with rare outcomes, you better have big data sources. And you’re not going to get
safety out of clinical trials. Real-world data is particularly
helpful to understand the safety profile,
especially when there is very limited information on
the background rates of some of the therapies
that we use in medicine. I’ve given two examples
here, one of them was a new first-in-class biologic that targets the interleukin 6 receptor inhibitor. We’d never used this molecule
anywhere in medicine before. The first use was in RA and
some of the safety concerns related to liver issues or liver enzyme abnormalities and hyperlipidemia. And post-marketing safety
commitments were being discussed and real-world evidence,
including claims data, was used to inform what a
five-year safety steering committee was going to look
at and the study design that would have regulatory implications. Maybe even more on point,
a different molecule that’s a targeted therapy
that effects the Janus kinase pathway currently approved
in RA was seeking a label expansion for a different
disease, psoriatic arthritis. And while the safety profile
concerns were somewhat known, there wasn’t a lot of information
about the background rates of those events in a different disease. So real-world evidence, including
health plan claims data, was presented as part of
the FDA package that was submitted in the dossier
for a label expansion. That was presented a month ago to the FDA and the Arthritis Advisory
Committee as part of the background to inform and
contextualize the safety for the label expansion for a new disease for a drug that was on the market, but approved for a different condition. And I’ve given the references for that. In terms of the conduct of trials, so we at UAB and about
20 to 30 other sites are working on a large pragmatic trial. This is intended to evaluate
the safety and effectiveness of the live Zoster vaccine among people age 50 and over on TNF therapy. This, frankly, from a
clinician’s perspective would be a no-fly zone because this
is a live virus vaccine. And the reason we don’t use
this much in rheumatology for people on immunosuppressive
or modulator drugs is the safety concern. If you’re going to have
something bad with a live virus vaccine in an
immunosuppressed population, it’s going to happen pretty quickly. Like in the first four to six weeks. It’s a very simple trial,
if you’re over age 50 and on one of five anti-TNF drugs, pretty much you’ll qualify. And the safety concern
is going to manifest in the first four to six weeks. So that’s the pragmatic bit. From the site’s perspective, each patient is done at six weeks. But one key question that
is of great importance is how long does this
vaccine’s protection last? So we have follow up
with a linkage to claims in EHR data for people as
part of the consent form. So that you don’t have to
keep people in this study to figure out, hey, what’s
the rate of shingles? What’s the rate of postherpetic neuralgia? And this pragmatic trial
is under an IND from FDA with the understanding that
it would result in a label change if the data that
accumulates is as hypothesized. This would be part of
that regulatory submission and the FDA accepted this study design. We’re using an electronic consent system to help with screening
and make that efficient and to randomize people in real time. The part that I wanted
to just spend a minute on that this electronic consent
includes an authorization both to obtain medical records centrally, sites don’t have to do that, and to link to external data sources
that might include health plan claims and EHR
data like that from PCORnet. As part of the trial’s feasibility, we did basically what David described just a few minutes ago. You have enough people in the
country that might do this, so we used Medicare claims
data and Medicare data, like everybody’s health plan data, knows who the doctors are
and how they practice, because that’s how they bill. So we found eligible patients
on therapies of interest, grouped them into their
treating physician, grouped the treating
physicians into their practice, and then we said, well,
which doctors have signed a 1572 form and do research
and then group that by the largest to the smallest offices. And that’s what this picture shows. The X axis is the number
of doctor’s offices that do clinical research that have signed a 1572 form that you would need to screen the Y axis, which is
the number of eligible patients for the trial. So depending upon what
estimations that you make regarding how many people will say yes, 25, 33, 50%, et cetera, that tells you how many sites you need. So we actually figured out
both how many sites we need, but then because you
know who the doctors are, we actually then can go
to those sites and say, hey, do you want to be in this study? At the individual sites now,
we are then using the same kind of approach to essentially
pre-identify people. Rather than the doctor
referring the patient to the study coordinator,
where he or she thinks about it one patient at a time,
that doesn’t work very well. The study coordinator can tell the doctor, “Hey, I’ve searched our
data using tools like i2b2, “or SHRINE, I already know
who the people eligible are.” And then the doctor is informed
by the study coordinator, this patient will likely qualify. So that really helps with screening. Electronic consent standardizes
what the trial is about. There is a cute little six-minute cartoon. It’s delivered the same
way to every patient, not with the vagaries of
what a study coordinator, or even what the PIs
would tell the patient. And then we have a knowledge review, which is really a quiz,
but it sounds better as a knowledge review.
(audience laughing) Do you get it, as a patient? And if not, then we bring
them back to the consent. This is all on the
tablet, this is not taking the study coordinator’s
time, to remediate that because we really want to get it. And there’s no paper data, there’s nothing to have to upload via a
paper CR case report form into an electronic data system. It’s all on an iPad tablet. And that same tablet is
used with its camera, so that if somebody manifests a rash that might be clinical shingles, that’s part of the study data that then gets sent to an adjudication committee to decide, hey, is this shingles or not? So electronic and digital images are part of the study repository. As part of the consent,
there’s a HIPAA authorization and electronic medical
record release form. So that’s where we, the central
studies coordinating center, can go get the medical records. We don’t bother sites to
do that because we want to make it really easy for the sites. There are other examples
and I have a reference at the bottom for a different
study we were involved in. This was a five-year head-to-head study of a new biologic versus a TNF therapy. And it had a cardiovascular
safety outcome. Part of it, there was
intended really to inform the safety, that’s really
the bulk of why it was done. But patients might not keep coming back and people switch therapies. So they were asked to sign a consent that allowed the pharmaceutical company as well as the clinical
research organization to have the identifiable information to go get the medical records. And I remember when my
study coordinator saw this. She called me and said,
“Dr. Curtis, we’re releasing “to a drug company patients
personal information? “A, they’ll never agree and
B, isn’t this an awful thing?” and I had to explain to her,
look, you’re looking for a heart attack and if you have a 3% loss to followup each year,
that’s probably going to exceed the event rate
and we can’t have that. That compromises the whole
point of this five-year, 3,000 person plus study and
that’s why it’s so important. It takes a bit of extra time,
but this is a nice example where the same approach
is being used as part of the post-licensure safety commitment. As part of this same notion
of making it efficient to do safety follow up,
Sentinel of course is very effective to find a number
of adverse medical events. There’s algorithms and claims in EHR data, we’re all familiar with
these sorts of things. But you can use the same approach not just to find outcomes using data claims or EHR by themselves, where
maybe you want to maximize specificity, but rather
just for case finding. Have an algorithm that
dials up the sensitivity, finds the universe of every
possible adverse event out there that you’re looking for,
but then because they signed this electronic medical
record release form, at the baseline visit, now you can go get all the medical records. You can centrally adjudicate
them, as we normally would do. And that’s how we’re working
it in this pragmatic trial. And nobody has to come
back for safety visits year after year just
to ask questions like, are you still alive? Have you had a heart attack? Have you developed cancer? Because you can find those
things in real-world data. And that helps minimize
loss to follow-up because patients don’t have to keep coming back year after year for safety follow-up. IRBs are okay with this, and
we and probably many in the room have done a number of
linkages with real-world data. Example language if
you’re going to collect Social Security numbers
from one of the studies we did where linked claims
data to a large registry, is shown as described. You don’t have to have
Social Security numbers, people are very skittish about
that for reasons we all know. Methods to use multiple
non-unique identifiers, sex, date of birth, worked quite well. You can use an honest
broker, if you need to. You can use hashing
algorithms and fancy computer algorithms to do this and that works fine. The last example in the last minute or so, is the post-marketing safety commitments. That if you really need
to study something rare, and this is an example from osteoporosis, where a first-in-class
biologic drug, denosumab, when we use it for osteoporosis,
it’s called Prolia. One of the things that was of interest was to study osteonecrosis of the jaw. The event rate for that is less than one out of 1,000 people. Claims algorithms by themselves do not work particularly well. So we at UAB contributed Medicare data, Optum contributed United Healthcare data and then there’s a European data source. Those three data sources that have some targeted medical record
retrieval in case adjudication, that forms the major bulk
of the safety commitment that the agency accepted as
part of studying the long-term safety of this first-in-class
biologic therapy. So this is perhaps among
the better examples I’m aware of where this
kind of data source, this is claims plus a
little bit of targeted medical record review, helped inform some of the regulatory implications about the safety of this biologic product. With that, I’ll turn
things over to Adrian. – Thanks, Jeff. Okay, I’m actually going to summarize with really three stories here. I think one of the
themes that you’ve heard on this panel as well as earlier, is that the real-world evidence is about the totality of data. It’s not a single study or a single trial or a single observational study. Really what we’re
talking about is creating a living textbook about
a medical product and/or patients that have experience
with that medical product. Whether it’s in formal research or whether it’s in
experiences in the real world. And if you consider an
example, and I’ll use my mother as an example, she
has premature coronary disease. She is very interested
in improving the health of people like her with
coronary artery disease. However, clinical trials is
not a convenience for her life. She cares for four grandkids and does an excellent job for us and she loves doing that every day. Now can you imagine her
coming in every week or every other week for a clinic visit where she has to park, where
she has to find her way to a site, a study clinic,
that’s usually in a back corner. And then wait and wait
and wait to donate blood to answer a series of questions. That’s unrealistic and
that’s what you hear all the time that people
are highly interested in being part of research,
but when it comes down to actual participation,
it’s very much the opposite. Now if you offer a clinical
trial, and I’ll use ADAPTABLE that we’re doing with PCORnet, testing two doses of
aspirin, where it’s actually at the convenience of a
participant doing the trial. Where someone is being
electronically identified, electronically contacted, consented, randomized and follow
up, you can have a very simple system where more people can engage and participate in research. Now it is important to
have the full context of what’s the benefits at risk, in the case of ADAPTABLE,
we’re focused really on the benefits of potentially less MIs or less death or the risk of bleeding. So it’s that benefit-risk
ration that you really have to understand across the
context of different people. Now often, in real-world
evidence, we’re talking only on the focus of the benefit. Most of the time we actually need to understand that risk-benefit ratio. But we don’t necessarily
need to think about it where we’re as worried
about the unknown unknowns. So as we develop study designs with, whether it’s NIH or other sponsors, people are very interested in doing trials that are simple, pragmatic
and we can easily define the types of endpoints
that are patient centered that are convenient to collect, that are able to be collected
in the healthcare systems, such as what Jeff was describing. Or with claims, but where
people have problems is starting to think about the
so-called unknown unknowns. Are there safety concerns that people have that so-called haven’t
been discovered yet, even though there has already
been a wealth of evidence that’s been developed
around the medical product. So in the case of aspirin, well, actually, there’s been over 100 years of evidence around the safety of that. So we actually know what
we need to focus on. Now if you started
thinking about that context of benefits and risk, then
think about other case examples. And I’ll use heart failure
as another example. Traditionally, and for good reasons, the focus of drug development has been to improve the longevity of life as well as to prevent a major complication such as worsening heart failure. However, patients also care
about how they do every day. Can they walk more? Can they do more activities? Can they have a better quality of life? If you think about that as an example, where a medical product gets on the market for heart failure and having
cardiovascular benefits, in terms of reduced death
or hospitalizations. But still, the interest from patients is, actually, can I walk longer? Can I do more? Can I spend time with my grandkids? That becomes harder to do if you start thinking about what would it take to do another randomized trial to extend the understanding within the
context of heart failure. Such as using devices that
actually measure accelerometer data to actually learn
about someone’s experience every day, every week, as
opposed to a six-minute walk. Which you’d have to come into
clinic to do on most cases. Or learn how their symptoms
are doing every day and the trajectory of that. In that context, you could
easily do a pragmatic trial having things that are
collected at the convenience of a patient, but then what
gets people worried about is the so-called unknown
unknowns in terms of, well, how would we
collect this type of SAE and report it in this fashion, et cetera. And so that’s one of the challenges. Now think about another case, diabetes. In 2009, there was
guidance that was released that actually really tried
to address the question that had come up as a concern
where medical products for diabetes were focused
on lowering hemoglobin A1c, but there had been concerns
about cardiovascular safety. There were a series of trials that were actually done in that context. And then now, actually what we’re seeing is actually trials showing real benefits in terms of heart outcomes. Now you start seeing that scenario and start thinking about, well, where do the benefits really extend to? Many of those trials were
in narrow populations. But then there is
populations that are related to patients with diabetes, so those, too, are at risk of diabetes. Or another example is patients
who have heart failure who could potentially benefit
from a specific medical product that showed that benefit in trial. Now to actually answer that question, if you were required to
do the same size trial that got the medical product to market and cost hundreds of millions of dollars, that would be a real challenge here. You actually have
potentially a great profile in terms of the safety, in
terms of the real world. You also have a great
profile in terms of what it does in its initial indication as well as cardiovascular safety. So really trying to understand
what’s the benefit to risk in patients who have related conditions. Heart failure or chronic kidney disease are two examples there. And then if you start
thinking about pulling all this together, in
terms of totality of data, Sentinel is a great example,
where it’s constantly evaluating safety of medical products as it enters the real world. So if you’re doing a
focused clinical trial in an expanded population,
understanding benefit and risk for a condition
that’s related and you also have a parallel system
that’s evaluating safety, putting that together as a
total package for real-world evidence can really be very powerful. So we are really having a
learning healthcare system and creating a living textbook that people are making decisions. Patients, clinicians, health
systems and regulators. – Great, thanks. I want to thank all of you for, what I hope has come out
from these presentations is a couple of things, at least. One is the wide range of potential real-world evidence use
cases or applications relevant to regulatory decision-making. So we’ve seen a spectrum
here in the examples and the concepts that
different panelists discussed and presented, and I
think the second thing is that this is all happening now, across this whole spectrum. And I’m going to a followup
question each of you in minute. And it’s going to be about
a practical next step that FDA and those working with FDA on real-world evidence can take to advance the kinds of use cases
that you’ve described. But I just want to frame, briefly, and connect this back to the
framework that we described in the background paper,
which are trying to build on. With FDA and all of you. There was a spectrum
of uses described here, ranging from pre-market studies
of new medical products. We heard from CTTI, from VERVE, I think, to some extent, too, about this notion of going from sites to
patients to patients to sites. Sort of flipping the trial design to make it more about
trials that can be conducted in real-world care delivery settings. And that fits with what you all heard in the first panel about this notion of trying to embed research and learning into the healthcare system. That requires this kind of
flipping and that is happening now through the kinds of
pragmatic clinical trials described here, Sean Tunis
and others in this room who have worked on a range of others. This is happening now. With randomization, but done
in a real-world setting. That is real-world evidence. We’ve also heard the other extreme of observational post-market
study commitments to address safety
questions, learn more the, addressing the unknown unknowns or getting a better idea and making the unknowns about the knowns
more precisely understood. Some of that can involve,
and often does involve, non-randomized observational studies. And what I think what Marc covered, that was kind of a spectrum between there. I think a lot of the ISPOR-ISPE principles could be applied, and
I think you had in mind thinking about how to do non-randomized real world studies effectively. Maybe going beyond just
post-market surveillance applications to other
kinds of label extension or modification studies and add a set of principles and ideas,
a set of principles and approaches for good study hygiene. If the kinds of principles and ideas that are behind all of the applications that you’re hearing about here
could be better understood, could kick the tires on,
could be a clearer part of guidance on developing real-world evidence for regulatory uses. You can imagine a potential
path to transforming how we’re collecting
evidence and with a lot of benefits in the process. So we seem to already be on this journey across this spectrum, what’s a next step? Based on each of your experiences that we could take to accelerate progress somewhere in this broad
range of use cases? In this potentially
broad range of going from between prospective and other
good observational practices, all the way to some
version of randomization? – Yeah, I’ll get us going. I think that we have the technology. We have the data. We have the knowledge
to bring it all together to inform decision-making. I think what we need from the FDA side is some structure and
some guidance on what kinds of information is
going to be acceptable. To me, the blending of
traditional RCT data on the one hand and
observational data on the other, in the middle there is the
pragmatic clinical trial. And when you think about the
changes in the regulatory guidance in Europe, where
they’ve invented a new concept called the low-intervention
clinical trial. When you look at that and
some of the aspects of it, it’s kind of a euphemism for
the pragmatic clinical trial. It’s for products on the market used in terms of typical clinical practice. And there are differential kinds of regulatory requirements
associated with that. Currently though, in the United States, we don’t have anything akin to that. I think some guidance along those lines will enable us to take that step towards greater use of the pragmatic
clinical trial design as an approach for real-world
evidence generation and use for regulatory decision-making. And I would argue that that’s
going to be the kind of real-world data that
will have the greatest level of rigor and comfort
level for all concerned. Because it still retains randomization, no matter what kinds
of methods you throw at the observational data,
you’re never going to get rid of all the
uncertainties associated with confounding and bias. But if you have randomization in place, miraculously, it seems to work. So I would like to see us
move towards greater use of the pragmatic clinical
trial design in the future. – Marc?
– Well, you know, if we’re going to have a learning
healthcare system, then it can’t just use
the data that’s thrown off by the healthcare system
just to assess safety. And we’re very comfortable with that. Pharmacoepidemiology has established good practices about how you
do signal identification, signal verification, that sort of thing. On the effectiveness side, though, there’s been a lot of concerns. Some of which has to do with the fact that the effects sizes tends
to be relatively small. Having said that, this data will be used by many, many different stakeholders across the healthcare system. And the question is, can we make it more credible to a wider
variety of decision-makers? And I think that just
as there were concerns 10 years or so, or whatever it was ago, regarding the randomized
clinical trial engine and it led to the International Consortium of Medical Journal Editors
to require pre-registration on if
they wanted to be published. And that the FDA, if you want to be considered for it, pre-register it. I think that the community
that’s using the same techniques that have been
used for 30, 40, 50 years for health services research
and pharmacoepidemiology, but are now applying it to
understand what are the benefits? And who benefits the most? I think we can move
substantially in that direction if the consumers of
those, potential consumers of that information, be it
journal editors or the FDA or payers or HT authorities,
require pre-registration. It’s not an absolute guarantee. At the end of the day, if people want to do fraud, they can do fraud. And fraud still happens today in all aspects of scientific endeavor. John Ioannidis talks
about this quite well. But I think that most people are of good will and good spirit. And I think that if you
say that the standard, what it will be, that you
know what, explore the data, but if you want to make recommendations, that you think are worthy
to change healthcare decision-makers ideas,
then you should follow some good procedural practices. And I think if we did that, over time, it will transform the entire culture of health services research
and outcomes research. – Jeff?
– From my perspective, I think that the FDA has been open, but it would be exceedingly helpful for an even greater openness
as well as an impetus, if not a mandate to go expand
the kind of trial endpoints that are going to be much
more informative to patients. Your mother wants to spend
time with the grandkids, how do you measure that? Could that be a trial endpoint? She doesn’t care what
her ejection fraction is, I suspect, and event
rates for heart failure hospitalization or mortality,
that’s rather crude. And that’s not really
what she cares about, so I fear we are doing trials that don’t really matter to patients, by and large. But when those kinds of outcomes, and assuming you can
quantitate them through the NIH PROMIS system or other
types of things like that, when that gets discussed or
vetted, as the trial’s outcome, I often hear, well, the
FDA doesn’t accept that. They’ll accept the SF-36,
and to my knowledge, at least in my circle of influence, there are exactly zero
clinicians who routinely measure the SF-36 at routine care visits. So you can get a label
claim for patient-reported outcome improvement using
a measure that nobody really even knows what it means and nobody uses it in the real world. So to me, I think that’s really where we need to push the envelope. We need the informatics systems to know where to stick that kind
of data using controlled terminologies or
ontologies, and not to have everybody doing their own thing. We need an expansion of
LOINC or something like that to know how to handle that kind of data. And PCORnet has made some
end roads in that regard. But to me, that will help us do trials that actually matter to patients. In a way that the data
can be more interoperable. – So I can see really three things. One is predictable pathways. So I think what we often hear is that there’s strong interest, the data’s there and in many situations, randomization is very
important and could be done. But what happens is concerns
about unpredictability in terms of evaluating benefit to risk and so therefore, it causes a lot of extra measurement of information
that’s hardly ever used. As an example, as clinical
study report may be this thick, the top line results, or it
could be done in a pages at most, and that’s people
make really their decisions. So predictable pathways. The second thing, and
it’s tying off the CTTI recommendations is right now,
there’s a lot of interest in how can you convert the
real world to the FDA world. So the example is
electronic health records. That doesn’t operate with CDISC. Clinicians don’t operate
in that way, either. So how do we actually
take advantage of what that is in terms of the
real world as opposed to trying to get everything
converted to something that is a standard that is not necessarily part of clinical practice. And then the third thing
comes out of actually taking advantage of the lifecycle
of a medical product. As you get further along,
in terms of lifecycle of the product, the
incremental information that would be gained
through doing the same study again, say, for a safety component, it’s going to be really less. I mean, it’s going to
have a hard challenge in terms of saying we’re
going to find something else. Especially in a setting where we’re having other evaluations that are
ongoing, such as with Sentinel. – Great, thank you for the comments. Now we have time for
a couple of questions. This was kind of a methods heavy session, given the range of uses and potential best practices and next steps. We have time for a couple of questions and comments on these topics, Bob. – Wait for the mic?
– Yeah. – Boy, am I surprised. This has been very interesting. Some of the kinds of uses people want to put are conceptually easier. So I grant that the data you
get from an electronic health record isn’t the same as what you get from a clinical study report. But in a randomized trial, I’m convinced that a lot of those conceivably could be credible. Survival, obviously would
be, but so would some others. Whether you had a heart
attack or not, like that. I think that that’s not so hard. The hard part, I think,
is when epidemiologic data is going to be acceptably
reliable or credible without randomization,
for an actual claim. That’s the hard part. And I guess I haven’t heard
so much about how to do that. One thing for me, is at least do it twice in two different settings. That was said before by Marc, I guess. But also the other thing
is to take some completed conventional trials that have been done and see if you can replicate
the result in the real-world setting using epidemiologic findings. Without knowing too much about
how they did the other trial, it has to be sort of blinded. And see if you get the same answer. We don’t even really know that. But that’s something
one could start to do. Every time there’s a
big post-marketing study of an anti-platelet
drug, do the same thing in your environment, see
if you get the same answer. – So, Bob, I disagree with you. There is a lot of evidence to suggest that results from RCTs and results from observational studies actually match up more than they disagree. There are very well-known
cases where they don’t. – [Bob] I’ve seen the Ioannidis things. – I know very well-known cases. Having said that, I think the issue is about the corpus of evidence. You never rely on one clinical trial. You should never rely on
one observational trial. If you have four different
observational trials done in four different populations, using somewhat different
methods and they all directionally say this is true, I gotta tell you, that’s as good as evidence-based medicine’s going to get. – [Bob] Maybe, unless there’s
some inherent bias in it. But you could also do what I said. Do your epidemiologic
studies where you actually have a study showing you the answer. See if you get it. – We may not fully resolve this. It does seem like there
is some agreement, though, on best practices, good
hygiene around how to do the observational studies, prospectivity, transparency and methods and all. I thought there were a couple of questions maybe over, okay, right here. And then Sean, and then I think we’re going to be out of time. – I just want to pick up
on Bob Campbell’s comment. I’m Bill Crown, I’m the
Chief Scientific Officer of OptumLabs and we’re actually
trying to get a project going in collaboration with MRCT and also working with
Duke-Margolis on this to replicate a whole series of clinical trials. The inclusion-exclusion
criteria of the trials, seeing if we can estimate the
same average treatment effects and then open up the aperture
and see who was actually treated and look at
treatment heterogeneity and under what conditions
can you replicate and under what conditions can you not? Really would be very interested
in anyone in the room, or in an extended group
that’s on the phone to get in touch with us
if they’re interested in learning more about
this and participating. – Great, thanks very much, Bill. Sean, and then I think
that’s probably going to be, Sean, you’re probably our last
question for this session. Or comment.
– Sean Tunis from the Center for
Medical Technology Policy. Another great panel,
several people on this panel and the earlier one
referred to the importance of outcomes that are
meaningful to patients being increasingly emphasized by PCORI. Patient-focused drug
development everywhere. The recognition is that
looking at benefits and risks of therapies is
going to be much improved by focusing more on gathering that sort of either patient-reported
data or other measures that are the patients say are
meaningful impacts on life. Probably people are familiar,
there’s various groups out there, like the COMET
Initiative and ICHOM that are trying to develop these kinds of harmonized patient outcomes sets. I’m just wondering if
anyone can make the link of how can we look towards
enriching the quality of real-world evidence
or evidence generated through clinical care
where we might actually have some meaningful
and consistent outcomes that are meaningful to
patients and not be limited to what’s easy to collect. But what actually is meaningful. – Well, that’s come up on both panels. As a real reason to pursue
some of these RWD efforts. Any quick comments? I see a head nodding. – Well, I’d just say incentives matter. Let’s just say if you put that as part of our USDs rankings for hospitals and health systems, I know we at Duke would really pay attention to it. (Mark laughing)
(audience laughing) And I think that’s the case
for almost every health system. So how do you actually
change incentive structure? – So I would just
comment, Cleveland Clinic instituted years ago in
their rehab thing area, collecting a patient-reported
outcomes questionnaire. And it was well taken
up because it actually helped the doctors take
care of the patients. Though if you do something
that helps a doctor do his job, or any
healthcare practitioner, do their job better, there’s
incentive for them to do it. When you tell people to
do things and they don’t see how it’s going to
do it, then you end up with what happens when you just layer in electronic medical record on a system. You don’t reorganize
how you do medical care. You end up with other
errors because people are just trying to get through the EHR. So I think there is fatigue among practicing physicians about
entering all the data in EHRs. There’s fatigue among physicians
about all the reminders and popups that they get
to tell them to do things. I had a doctor a couple years
ago who I went into see, and it’s not my doctor anymore. He’d ask me a question and then. A minute and a half would go by, then he would talk to me again. We have to re-engineer how
healthcare is delivered. The EHRs of today are an
enormous step forward, but they are not the final
place where we need to be. Some clinics have gone to the fact that they have someone
who just records the data and the doctor doesn’t have to sit there and record the data into the EHR. I don’t know if that’s
economically feasible or whatever. And maybe a technology will improve that. AI will listen to the conversation and automatically put
things into a record. But asking for perfect information now or near-perfect information now, will cause as much problems as it’s going to solve
in the next five years. – All right, did you have
a last comment, David? – Yeah, I just had one final comment on this discrepancy
between clinical trial data on the one hand and
real-world data on the other. I would argue that we’ll know we have made tremendous progress when we stop asking why the real-world data don’t
reflect the clinical trials and we start asking why
the clinical trial data do not the reflect the real-world data? – (laughs) It’s back to
that flip the trial concept. All right, I want to thank our
panel for a great discussion. Thank you all very much.
(audience clapping) Now are going to adjourn for an hour. So we’ll be starting up again at 1:20. If you have any questions
about restaurants, food in the area, please
ask at our reception desk. (people chatting distantly) – [Man] Well, you know, I’m from a CRO, but I’m a health economist by trade. (people chatting distantly) Some of our trials have
been five year study image. (speaker drowned out by conversations) if you know how to conduct
that in population X, you know how to conduct in population Y. – [Man] Right. (people chatting distantly) – [Man] Weren’t you a resident for Rachel? – [Man] I sure was. – [Man] Nice to meet you. – [Man] Yes, I remember. A long time ago, that’s right, I still stay in close touch with (mumbles) who I’m sure you know. (people chatting distantly) Yeah, I still actually do a
fair amount of stuff with. (speaker drowned out by conversations) It was my classmate, and
he left and came back. So he’s on faculty there. – [Man] Were you a med student there? – [Man] I was a med student and a resident and I got my MPH there. So I finished all that and
graduated with the MPH in ’99 and then residency in
medicine in ’02 and then left and went to do some fellowship
and then the grad school (mumbles) south and northeast. (man speaking distantly) About 12 years, I did some
informatics work at Stanford and did epi training in Boston, so yeah, kind of went all
over, love the Midwest, yes. – [Man] So I worked at PCORI (mumbles). – [Man] Oh okay, great. (man speaking distantly) So I’m the CO-PI of the
PPRN of Arthritis Power and have a demonstration
project that’s bringing together three CDRN’s data and five PPRN’s data and it’s a little bit
like herding butterflies. But it’s going and the
demonstration projects. Yeah, it’s intended to
use the infrastructure for something useful and important. So it’s going well, but
finding all the hiccups and speed bumps along the way. But I think all the
demo project’s expected to uncover and unearth (mumbles) in large part with (mumbles). – Yeah.
– Good, so what’s your role with PCORI? – [Man] I’m the Director
at one of the clinics. – [Man] Okay. (man drowned out by conversations) Yeah, we’ve done a number of things. Some successful, some not as successful. But yeah, our registry
though is doing very well. We’ve started from zero and
we just broke 11,000 people, mostly with RA and autoimmune diseases. Yeah, yeah, so we’ve been
making great strides. – [Man] I have a question about (mumbles). – Yeah.
– You talked about (mumbles). (people chatting distantly) – [Man] So hardly anybody
ever gets the rolls from Medicare, as you know. Now this enrollment rate
is for commercial claims, of course it’s super high for
reasons that you know well. I guess my premise is that
there is no one perfect data source, so consent
people that you can follow them wherever it is they
are, in whatever data system that they may be bound in over time. So if they’ll keep coming back when (mumbles) your study visits, fine. If you can link them to commercial claims, there’s three or four big players that are going to have a lot of your data. You could pick your trials sites based on where those sites have
lots of representation. And in fact, we looked at that
for the Zoster vaccine study. We’re going to pick our trial sites because we know we have a high penetration of company X and that we
can keep following people. But it’s really with the premise that there’s never going to be
one perfect data source. But if you could bridge across them, because people gave you
some identifiers and consent at time zero, you had one or
at most two clinical visits, got their permission to
follow them over time and whatever data source
you have access to. To me that’s the way to go. Because then you don’t have to choose this data source and then they change jobs and change insurance and we lose them. But you can really follow
them in a much more continual and longitudinal way. That is really helped by consent and some identifiers at the outset. – [Man] Yeah, well it was good to see you. – [Man] Nice to see you. (people chatting distantly) – Oh hi, how are you?
– Nice to meet you. – [Man] Yeah, nice meeting you. – [Man] You’re still good to talk at 2:30? – [Man] Yes, so do you
want to just pop out to where the waters are
and stuff like that? All right, all right, good, great. – [Man] Great, I’ll
catch up with you then. – [Man] Okay, thanks. (people chatting distantly) – [Man] I did the breath check. (people chatting distantly) – Nice to meet you.
(people chatting distantly) – [Woman] (mumbles) Nice
to meet you, all righty. (people chatting distantly) – [Man] We have like five more minutes. – [Woman] Okay, all right. – [Mark] All right, I’d
like to ask everyone to head for their seats. We’re going to try to start again in just a couple minutes, thank you. (people chatting distantly) – [Man] I’m going to be at the far end. – [Woman] We’ve been assigned seats. – [Mark] All right, good
afternoon, everyone. I’d like to welcome you all
back to this afternoon’s session of our real-world evidence event today. For those of you who are joining online, just a reminder that if
you have any questions or comments along the way,
you can send them to us at [email protected] That’s [email protected] and we’ll get those questions or comments into the discussion if we can. And I hope many of you are
following along on Twitter. There’s been some good discussion, Just use the @dukemargolis
handle and #RWE. So for our session this afternoon, we’ve already talked
this morning about many of the challenges and issues in developing real-world data and applying
it in fit-for-purpose uses for real-world evidence development. And as you heard, real-world evidence is already in development and use. Including in some issues related
to regulatory applications. What we really want to
focus on in this session is some of the industry perspectives, some of these perspectives
from the companies, the organizations that are actually involved in the development
of real-world evidence. And in a position where
it could actually be submitting such evidence
for label modifications, for post-market safety information. Some of the topics that
have come up earlier. Even potentially for
practical clinical trials or other types of real-world evidence in the pre-market setting. So we want to hear about
how that experience is going with an eye toward some of the scientific or infrastructure or
cultural or other barriers or issues that might get
in the way of continuing progress on real-world
evidence development and use. And we’re very pleased
to have four panelists with us today that have
some extensive direct experience related to real-world evidence and related to regulatory applications. Amy Rudolph is the Vice President and Head of US Pharma Health Economics and Outcomes Research
and Early development at Novartis Pharmaceuticals. Jacqueline Law is Vice
President and Global Head of Real-World Data Science at Genentech. Symantha Melemed is the Global Product Team Leader at Eli Lilly & Company. And Joanne Waldstreicher, Chief Medical Officer at Johnson & Johnson. So as in our previous panels,
we’re going to start out with some opening comments
from our panelists and have a bit of discussion. This is a panel where
we want to make sure, I think we’re going to have
a relatively short comments. We want to make sure that
we’re hearing from you all about issues, concerns, opportunities that you see in these real-world evidence applications for regulatory purposes. So Amy, please go ahead. – Just to start, thank you
so much for the invitation, Mark and colleagues. Good afternoon, I’m Amy
Rudolph, as Mark mentioned. We took a little bit different tack, we wanted to have more of
a practical discussion. So I don’t have the clicker, maybe if you could jump to the next slide. What we thought might
be useful is to share some perspectives on some
key tenets to consider as the guidance development is underway. I think we call agree on these tenets. And they were elegantly
captured in the white paper and throughout the discussions today. We talked today a little
bit about the environment, the complexity of the environment
demanding new evidence. I think we can all get behind that. That really spans the
entire data continuum. I think when we think
about data collection, we’ve talked today a little
bit about reproducibility of data, we’ve talked about some primary and secondary data sources and the fact that we need to bring
together data compendiums. On this, I’ll note that
we haven’t talked a lot, although it was captured
in the white paper and maybe we’ll touch on it
is, one of the neat things about capturing real-world
evidence and using that to compliment existing trial data sources is it can be cost savings. A lot of this discussion comes
up around pragmatic trials. And certainly when we think about the need for large data compendiums, we don’t want to lose the fact that we could save costs. So we want to control the compendium but have a robust compendium
for offering data forward. In terms of tenets, again, these are just, it’s certainly not an exhaustive list. I think we could all agree to that. But just by way of what
kinds of elements might be critical for the guidance
to include and likely will. Certainly bias management has come up. One thing of note and it’s
been touch upon today, but just maybe to state
it a bit for the record that I think we all recognize,
but again it’s important to state that bias mitigation
cannot be absolute. And I would argue and I
think others would as well that bias mitigation
can never be absolute. But I think that’s
important to keep in mind and we’ve talked about EHR and some data, missing data and some other challenges. We can mitigate the
bias, but not completely. I think we need to, if we can, set boundaries for what is good enough. Marc, you talked a little bit about that, but also around what kind of primary versus secondary data
sources would be acceptable. And are there boundaries
for what’s good enough? And what is outside of those boundaries? If we can get some information
around that I think that would help us all try
to bring forward the best robust packages that would
fit within those boundaries. We talked a lot about
database suitability. If we can have some
directionality around that, if there are databases or
approaches that are just not suitable, again, it’s
around the boundary question. I think we should state
those as clearly as we can. And the last point is
around patient-centric data. This was raised, it was
touched on previously by David. The concept of the collection of lots of different types of data. Adrian talked about his
mom and her experiences. Sally, great discussion around fatigue, I thought that was a supreme example of the different types of data. And the use of sensors,
we’re becoming more and more intelligent about how
do this in an effective way. But I would argue
nationally, we don’t know how to handle big data, sensor data. For example, I think the
value framework organizations are struggling how to integrated PROs and even sensor data into that framework. I think it also is applicable here that we’re all kind of
struggling how to do that. But if we want to have an intimate look at the patients’ lives,
this is one way to do it using different types of data and we need to be able to fully integrate that. I captured here
adherence/persistence also, fundamental as well to
the patient experience. Turn it over to you.
– Thank you. – Yep, thank you. Yeah, thanks again also, Mark, for the invitation and
have the opportunity to share industry perspectives here. Next slide, so I just have one slide. I think there are many, many factors that are driving the
interest and opportunities of using real-world data to really support a broader healthcare decision-making. First of all, it is the improvement in the quality of real-world data. The improvement in the availability as well as the speed of
these data become available. And also I think this
morning we heard about maybe the relevance of the data. Clinical trial data look
at certain endpoints, but in the clinical
setting, some endpoints may not be relevant any more. So I think that’s fundamental, in terms of driving this trend forward. But also with the advances in medicines, diagnostics and technology,
in the area that we’re developing personalized
medicine, precision medicine. The way that we develop
drug need be very different than the traditional
way of drug development. So coupled with the increasing
drug development costs and timeline and also
the pricing pressure. I think all these different factors really drive the industry to think about how to use real-world data
to inform decision-making. So we’ve heard that there
already opportunistic use of real-world data to
information regulatory decision. Including the rare disease setting as well as the post-marketing safety surveillance. But how we move forward confidently
of using real-world data to inform broader
healthcare decision-making. So there’s some ideas how we can move this from a concept to practice. So first of all, we heard
a lot about data standards and how we can really set the standards of collecting data, what kind
of data need to be captured. Also the quality assessment
and the requirements. To Amy’s point, how good is good and what are the boundaries? We’re not expecting real-world
data to be monitored and maybe not as clear
as clinical trial data, but does that matter? Is that going to
influence decision-making? And then endpoint definitions, definitely is a big
topic in real-world data. It’s easier said than
done, in terms of defining the relevant endpoints in
the real-world setting. And it requires a lot
of collaborations across the industry with FDA and
academia to really think about what is really the relevant endpoint when we look at the data
from the real world. And then the other thing is how we can protect data and patient privacy. I’m so glad that this morning, a lot of conversation on the governance. So it’s really fundamental for us to protect data and patient privacy. So one of the requirements when
we look at real-world data, is there going to be a
informed consent required? What are the HIPAA
requirements, et cetera? These are the things that need
to be defined and clarified. And then going down the line,
when it comes to submission, we are all very familiar with
the submission requirements data package required for
clinical trial submission. So in the real-world setting, are we going to think about audit? Where there’s a source data
verification requirement. So these are the more practical things that sponsors would need to think through from the very beginning of
the clinical development, if we go down that path
of using real-world data to support regulatory use. And I think one thing
that is really fundamental in terms of this industry is
we need to change our mindset. Using real-world data to
support regulatory use is an innovation and driving innovation, we need to foster the right
environment and culture. So right now there are
a lot of hesitations even just to think about this path because of the uncertainty and
also the potential impact on timeline and the
risks that are going to come with using real-world data. So having the opportunity
to have early input from FDA, interactions with
FDA on the drug development program using real-world
data would really help remove some of this
uncertainties and engage different stakeholders to have
this conversation together. And then finally, it’s to
really have the opportunity to have pre-competitive
sharing of use cases. Again, we are in this journey of driving an innovation in this industry. So finding ways that we
can learn from each other, be it a successful use
case or a failure use case, I think we can all learn from
how to move this forward. – Great, thank you.
– Very good. I don’t have any slides,
but just a few things to add on to what my fellow panelists have so eloquently described. Some of the areas that
we’re really interested in using real-world evidence is certainly on the regulatory side,
but then also moving it earlier into the lifecycle
program of the drug. When we’re making these
trade off decisions between a traditional
randomized clinical trial and a pragmatic or observational study, the more clarity we have in
terms of where that will fit in to our submission activities,
global registrations and other clinical
development for the compound will really help as we look
at developing these compounds. Some of the areas that
we’re really interested in is around using real-world
evidence for dosing, for historical controls,
even understanding biomarkers as we’re starting to get these high-quality linked data sets, where we feel good about the
quality of the biomarkers that are there and the level
of detail the laboratories that are performing the analyses. Potentially using it for control arms for rare populations or in oncology, where I spent the majority of my career, using that where we
have single-arm studies to understand how the drug’s performing in a rapidly evolving
scientific landscape. For post-marketing
commitments and then of course for line extensions, which has already been extensively discussed. So as we start to think
about the data ecosystem that we would be bringing forward, we certainly remain
committed to randomized clinical trials, but what
we’re looking forward to is guidance from FDA and then as this room and people on the web as
well evolve the landscape, is the data ecosystem that will exist between observational trials
and pragmatic clinical trials. And then of course from the
randomized clinical trials that have provided the
foundation of the majority of work that we’ve done
to bring new medicines forward over the years. A few things I just wanted to bring up in terms of barriers, as I
think you alluded to it nicely, is that this is an innovation. And we have generated a group of folks that are used to doing
randomized clinical trials. And so as we think about
the regulatory science, How do we do the submissions? How do we do audits? I think the people and processes will need to evolve at the same time. And then also, I just wanted to call out that we do, we believe,
have a unique opportunity in oncology, we’re certainly
interested in real-world evidence across the entire spectrum. But I think in oncology, we
have key stakeholder alignment. We have a very engaged
community from the advocacy community to the ASCO,
and other unifying bodies. From pharma and others,
we have some really great examples of high-quality
data sets that we can use. And the rapid pace of change in the space, I think we all feel it, whether
or not you’re in oncology or not, it’s unbelievably quick. So I think we have a really
interesting opportunity there to look at pilots and
others moving forward. So I think we’re excited to get started and we look forward to
the conversation today. – [Mark] Great, thanks Symantha. – Thanks so much for the opportunity. I think what we saw in the white paper, which was excellent, and also from many of the speakers today is that rigorous real-world evidence really helps provide insight into questions that are difficult or infeasible or cost prohibitive. And can really help drug
development or post-marketing, from both an efficacy as
well as a safety perspective. I think what we have
the opportunity, though, as an industry to really help contribute to the goal of a learning
healthcare system. And I think this is where we focus and where we want to focus from
a public health perspective and set the bar higher for all of us. And I think we would all benefit as a public health system if we do that. So the themes that I
want to talk about were being more transparent with our data, our analyses, our methods and our results. And really trying to think about the learning healthcare system. I was given the assignment,
I was volunteered, to talk about safety
today, but I don’t want to let the opportunity go
by without mentioning a couple of points about effectiveness. I think you can’t say enough about the potential for
predictive modeling. It’s great when we can
use real-world evidence and real-world data to
understand the generalizable benefit and risks, but
really there’s incredible power to come if we can
use predictive modeling. Both from traditional sources
of observational data, as well as new sources that we have and that you all know about. Patient types of data,
people types of data, that we can then put in
for predictive modeling. Because it’s all about, in the
learning healthcare system, it’s all about maximizing the benefits and minimizing the risks. Right now it’s difficult for us, even if we believe that we
have good predictive modeling, it’s difficult for us to
be able to discuss this, and we’d love to be able
to work with regulators to be able to do analyses
that meet the rigor that they would agree with
and that we could then talk to healthcare providers and patients to be able to maximize the
benefit and minimize the risk. I also don’t want to let
the opportunity to go by, and this was something that
was raised by Laura Esserman, we really believe strongly in platform trials and master protocols. I think the more that we can do to have similar protocols or the
same protocols, the same infrastructure, so we’re
evaluating products on the same basis with the same
endpoints and the same rigor. Even if products can’t be
evaluated at the same time, but if they can be put
into master protocols or platform trials as
they become available, I think we all benefit from a
public health by doing that. And of course, it raises
the bar for us as industry, but that’s a good thing. It forces us to go higher and better and for more transformational therapies. And I think everyone benefits. So I wanted to talk about safety. In terms of safety, I think that we could think about a future where
we could work together to further understand the
safety profile of our products. Going even farther, but
again, I want to keep those themes in mind of
transparency, inclusiveness, using the best available
methodology and analyses. So as we all know, the
safety of our profile is driven by clinical trials, of course, especially pre-approval. But of course we have them post-approval. Then post-marketing
adverse event reporting or spontaneous reports, as we call them. And then if a signal is observed, an observational epidemiology study. And for retrospective
observational studies, rather than randomized studies, the white paper says, and Marc Berger stole some of my thunder,
so thanks for that, (audience chuckling) talked about the importance
of pre-specification of the analysis plan, the
strength and diversity of the real-world data
and the reproducibility across different databases. This is something that we
could not agree more with. We think that there could and
there should be right now, much more rigor in insisting
on pre-specification. And I would go further and
say even pre-registration of these types of protocols
and analysis plans for observational studies. Protocols for observational studies can be registered on We have done that, they can be and they should be registered. And although the system
is set up primarily for clinical trials, there’s no reason why we, journals, others, can’t insist that protocols be registered
before they’re done. There also a registration
site for meta-analysis protocols called PROSPERO. We also have to be sure of the strength and diversity of the data. And there’s no reason
why we shouldn’t insist on multiple databases and
evaluating the reproducibility and the rigor of the data that’s generated from real-world evidence studies. I think we could go even a step further. As I said before, the sequences, putting aside clinical trials, of course, which is very robust for
safety signal detection. We now live in a world
where we have spontaneous, as we all know,
underreported adverse events as well then looking
at observational data. So I’ll start with the observational data. Observational studies, like Sentinel, which has been incredibly valuable for understanding safety
and for public health. Now as we’re moving
forward and we’re getting more experience with
these types of analyses and we get access to even more databases, not just in the US, but
globally, worldwide, and we’re sharpening and
developing our methodology, might there be an opportunity not only to wait until we see a signal
to do an observational study, but to proactively look
at observational data for signal detection in
addition to spontaneous adverse event reporting? We’re looking at methods to do that. To proactively look at observational data and mind you, when we do that, we see not only our own drug data, our own treatment data,
but we see everything, all the different
treatments, all of the data. Is there a way in the new
world that we’re living in to proactively look at those
data to generate signals. Which could then be confirmed or refuted by doing an epidemiological
study, such as Sentinel. Since we’re talking
about observational data and we’re talking about
safety, is there an opportunity now as we’re getting better understanding observational data and
I love the discussion from Laura Esserman
about seeing and looking for adverse event reporting
within their health system. As we’re doing more of
that, can we revisit spontaneous adverse event reporting? Which we know is severely
underreported and not perfect. Of course adverse event
reporting is critical and will continue to be
critical for years to come to understand the post-marketing
safety of our products. But since we’ll see
everything in real-world data, and we can get better
and better over time, what can we do better or differently? And I would throw out, what
can we even stop doing? For example, at our
company, we literally spend millions of dollars and I counted, we have hundreds of people working on non-serious spontaneous adverse events. Not just reporting them and
entering them into a system, but reviewing them and auditing them, reporting them, summarizing them, getting ready for health authority audits. There are hundreds of
people looking at that. I just asked my group over
the past couple of weeks to look at a few large
products, just as an example. And I asked them, were
there any label changes, based on non-serious adverse events in the past few years? Now of course, this is not comprehensive. And I’m sure that there are
label changes all the time, but in this recent review
that they did for me, there were no label changes
based on non-serious adverse event reports
from spontaneous reports. So the litmus test that
we should take forward is are we advancing
public health by all this enormous effort that
we’re devoting looking at non-serious adverse events? Something to think about in the future, whether we could look
at observational data, health data, data in the
University of California system. Other ways, PatientsLikeMe, other ways to look at non-serious adverse events which would be better from
a public health perspective. We could also think
about looking at serious and even designated medical events. The adverse event
reporting system has played an important role, I
would say critical role, in defining the safety
profile post-marketing. But could we do even more and even better? As the learning healthcare system, in addition to spontaneous reports? Could DMEs be flagged? I know everyone hates
flags, but if anything’s going to be flagged, could
those somehow electronically, all of them, be captured
so that we immediately in real time get access to the
broad amount if information surrounding those designated
medical events and get better follow up and more
complete reporting of events? And I would say maybe even
faster reporting of those designated medical events so
we could label them earlier and prevent more harm in the future. And be much more efficient and complete, again, taking that public health lens. So just putting it all together, my hope is that we can work together over the coming years
to define how we can be more transparent, work
together in a more open way. We’re getting broader and
deeper into real-world data. And as I said, we see it all. As my group looks at the real-world data and looks to understand our product, we see the data with all the products, all of the outcomes. Is there a reason we need to do this separately and in silos? Is there a trusted third party or a public-private partnership we could work with together to look
at all the data together to generate signals, to
understand and evaluate signals. To do this together on
behalf of public health. Could we do more predictive modeling to maximize the different,
minimize the risks, then revisit the data and
see if we’ve had an impact. Have we decreased the adverse events that we’re seeing because we’ve stopped using the drug in
certain types of products who have the highest risks? And we see collectively if
we’re accomplishing our goals as a learning healthcare system. Could we even pre-define
our protocols, post them and actually make the source
code available publicly? Could we do multiple types of analyses as we do one protocol, do
many sensitivity analyses and many different types of analyses. Make it all available publicly. And as I said, do it all together to empower the learning healthcare system. Thank you.
– Great, thank you all. Clearly, as we’ve heard earlier today, a lot of potential for better evidence. In some cases, new kinds of evidence or potentially transformative evidence at a lower cost than what’s been available with existing systems and even existing uses of real-world evidence to date. But I also heard in the comments, some concerns about uncertainty
about how to proceed and the risks of
incurring additional costs on top of all of the existing
regulatory requirements. And I want to follow on that a little bit, to try to take us again
to some practical steps that will get us to
that longer term vision. And a number of you
mentioned areas in which more clarity, more guidance from FDA would be helpful, like around submissions and like around early input. Although, other Jacqueline,
Jacqueline Corrigan-Curay, mentioned that FDA wants
to hear from you early, about real world uses. So we’ll come back to the FDA issues, but I want to start
with some of the issues that Joanne and others mentioned around what industry can be
doing differently to help. Or what industry needs
to do differently to help hasten these efforts along. And maybe you could start
with whether you all see any cultural, or are they
more technical barriers, to some of the recurrent themes here? Transparency, prospective
definitions of protocols. Something that’s done a
lot in the clinical trial context, but maybe not so much here. Reproducibility, via data
sharing, steps like that. Those are not the way
industry practices work today. What’s standing in the way of that? If you all seem to agree that
those are important steps for advancing real-world evidence? – Maybe I’ll start. I think, and please add perspectives, there is a tremendous amount of rigor that happens and it’s just not made, it’s not shared, it’s not exposed. The what I’ll call internal
rigor around making sure it’s pre-specified and we
all have forums that do this, work in a very collaborative,
cross-functional way. What a step forward would
be that we invite others in, from outside of our bubble, I think, and make it even more transparent. So it’s not that these things are not happening, I would offer. In terms of transparency,
pre-specification, but it’s not happening in a broad way. And it’s maybe not happening in a way that is fully collaborative. I don’t know if you guys would. – Yeah, I agree very much with you. I think rigor, definitely
is a key component to really build trust
internally and externally. However, there’s no
platform or opportunities for us to broadly share
that and how we are thinking about rigor and quality. So I think that would be really one step that we can take forward
to share our experience, our methodologies and
bring all these together. – A lot of that, it can
be done pre-competitively. Because it’s a specific
research question around the specific medicine that ends up being the competitive part of it. But a lot of the sharing
and standardization and all of that can be done
in a pre-competitive space. – So let me just add a couple points. I think we face the same cultural issues. And when I say we, I
don’t mean just industry, I mean sponsors or people or
groups that do clinical trials. It’s not just industry. We face that with clinical trial sharing, as was mentioned earlier. And we’ve found a way, we
found a path to overcome that. We started working with
trusted third parties. We work with Yale and YODA
and many of the companies work with CSDR with WELCOME and Duke. So we found a way to start doing that. It all started with registering
clinical trial protocols. And as many of you have
seen the publications, that wasn’t widely done
until it was required by the journals, by the medical journals. As soon as they set the
guidelines that they won’t publish a study until
studies are registered on,
there wasn’t that much and then there was the hockey stick up. But we found a way to do it. We found a way to post
our clinical trials. And we are also are now finding a way to share our clinical trial data. Getting around the issues and working very carefully around consent
and privacy, et cetera. So I think that there is a path forward and we can work, especially
after conferences like this and some of the meetings
coming up with IOM, there is a path forward to do that. And we can start, but it doesn’t
just require us to do it. I think it requires other
colleagues in academia and other parties that
do observational studies. We have to all be in this together. – I think there needs to be
this mindset, is not there yet. And we are all learning together. We are sharing experience and methodology, not because everything has to figured out. The data quality is not all there yet. But we are sharing that so
that we can learn together along with the data providers and others. And okay, how should we improve so that we can eventually get there? – Yeah, it sounds like a great time to add to the platforms for supporting this area. I like Joanne’s analogy to what’s happened with clinical trial transparency
and rigor over the years. To build on this, several of you mentioned this notion of platforms. So Joanne talked about
it, fairly extensively, around post-market safety
active surveillance using observational data
on pretty much everything, it sounded like, from your comments. But Symantha, you mentioned
other potential platforms or use cases, too, so I
wonder if I could push you all on some additional specific examples. Maybe more on safety, if you want to expand on that, Joanne or others. But Symantha, maybe I
could start with you. You mentioned that
oncology might be an area that’s primed to do this,
could you say more about what this kind of pre-competitive platform approach would look like? It sounds like it fits a lot with what Laura Esserman was talking
about this morning. – Yeah, absolutely. There are some advantages
in how data is collected in oncology that has
helped us move quickly. One of the things that certainly from that that we’re interested in is the amount of biomarker data collection that’s now routinely being done in oncology. And as we think about
using real-world evidence, not just to register
products on the back end or in the later part of their lifecycle, start to understand patient
populations that they work better or don’t work as well in. And do that based on the
high-quality biomarker data that’s being
collected at the same time. In addition, we’re often relying on either making big investment or
clinical trial decisions or even making decisions
to go to regulators with single-arm clinical
trials where we see big effect sizes relative to what we think
of as historical controls. Of if we’re seeing a
biomarker that’s in a very rare patient population,
where traditionally you would be testing 100
patients to find four, are these tests are routine
enough and that we feel confident enough in the quality of them that we can go and find
some of those patients and start to see the benefit there. Not to double-click on
something that I’m a big geek about, but understanding then how that 4% of patients performs
relative to historical control when the field’s rapidly
advancing really matters. So we don’t know if we’re
seeing a big advantage, because prognostically they’re better, as they often haven’t been studied. And there’s often ways that we can do this where we pre-specify. So I think in oncology,
we just have a massive amount of relatively high-quality data that we’ve got some really good curators that are working on, it’s not perfect. We share all of the
same concerns that were keeping folks at night in the morning. But I think that there’s
some real advantages that oncology provides right now. – Some other platform
use case opportunities that you all would like to talk about? Some other specific opportunities? – Well, even in situations when we are not able to do a clinical trial, I think then how can we leverage
real-world data to identify the patients and maybe also
looking at off-label use and seeing where the activity
is and treatment benefits are. And the order and biomedical needs. Or use real-world data to
create some synthetic control arm to, as a contemporary controllers. Symantha mentioned that the
field evolves so quickly. So looking at literature
and historical control would not be relevant any more. So I think these are many
uses of real-world data that we can think of in the
practical drug development. – Let me extend it to
more common situations. Someone this morning
mentioned the diabetes medicines which require now
cardiovascular outcome trials. As I look back several years, I think, I can’t even count the number of companies that have done their own
randomized placebo controlled cardiovascular outcome
trial, which is great. But in the future, might it be also good or instead, could we work together through a trusted third party that runs the study? So we’re looking at multiple products in the same study with the same endpoints and the same rigor and potentially even looking at combination drugs. If you think about
taking any of those drugs that have shown cardiovascular
benefit for diabetes and now doing a combination study to see if the combination is better
than each individual alone for a cardiovascular outcome trials, you’re talking about, I’ll be on Medicare. I told someone this morning, I’ll be on Medicare by
the time that’s done. And it’ll require hundreds
of millions of dollars of investment, et cetera. And isn’t it best for everyone if drugs are compared in the same trial framework and we can right off the bat look at them together and also separately. – So what’s getting in the way of progress on these kinds of platforms? It sounds like some of
this is happening now, you got the ingredients. What would accelerate progress? – If I could add?
– Uh-huh, please. – I was going to add that
from a platform perspective, I’m going to take it up a level. You kind of touched on it. This is actually happening. There’s advisor groups,
PCORnet advisor groups. We’re all coming together. I might argue, though,
it’s still a closed system. Why can’t we pull more patients and payers and other individuals
within the ecosystem, to borrow your word, why can’t
we pull them in together? And that’s happening a little bit, but I think we need to do that
in a much more concerted way. Because I think to your
question specifically, that is what’s going to
take us to the next level. Where you’re tackling big
health care challenges together, pulling folks
from the FDA together with the table, with us, with patients, with the payers in the system so that we can all tackle it together. And meet the collective needs
of ideally the patient first, but then all the other needs as well. – [Mark] Other thoughts on acceleration? – I do think regulatory
clarity is a huge piece. I know that that’s, I mean,
reading the title of the meeting, it’s a little
bit like Captain Obvious. But I do think for us as we make trade-off decisions. And there’s a lot of
excitement and there’s a lot of enthusiasm and
there’s a lot of interest in finding what pilots do we invest in? How do we work? I think this is an area
that is really neat, I worked in the biomarker space as well. And I feel like there’s a
lot of collaborative spirit and a lot of energy, but
we’re trying to figure out where’s the best place to aim it? So that we can bring our medicines forward faster and better? And I think when we
start to get some of that regulatory clarity, and even hearing like, say, hey, here are our ideas, having FDA say, “This is what
we’re interested in as well.” And then having that say, okay,
now as we start to register a medicine, this is how we can make that progress that will really help. I had promised we would
get to a discussion of opportunities for regulatory clarity. And having been on that side of FDA, there are an awful lot, I mean,
this is complicated stuff. – Of course.
– Health care and medical products, so there are
a lot of areas where FDA could provide more guidance
or input or feedback. Anything that you would like
to particularly highlight? I have some useful items,
Jacqueline, on your slide, that went into the tenets
that Amy described. One of the things that you mentioned there that I would like some comments on, but again, feel free to raise others, is this issue of how
much bias is tolerated? Or how much data imperfection and result? Consequential result
imperfection will be tolerated? That’s something that
several of you brought up. But I appreciate hearing about that and about any other particular areas where more regulatory guidance or engagement would be helpful. – Yeah, so maybe I can
share our experience working with oncology EMR. So obviously, the
mortality is not regularly captured in the EMRs,
so we do need to find a way to supplement the EMR data to come up with the mortality outcome. So at the end of the
day, do we expect 100% completeness of the data? Or how much missing data can we tolerate? And really it depends on the context. If you are thinking about
using that outcome data to support a regulatory
decision for a treatment, that depends on the treatment effects. Some treatment you would
be able to tolerate a higher degree of missingness. And if the treatment effect is huge. And this is probably the area that we need to go a bit deeper into
the technical side on, okay, what kind of treatment effects are we looking at in what setting? So then the data quality
depends on the context. And I think this is an area that we would need to collaborate
across, because this needs to be a common understanding
and agreement on okay, how do go about to
collect complete data? And also with FDA guidance on
that as well and agreement. Because it’s very challenging if we do not have clarity or
at least an alignment on, okay, how do we approach this problem. – And then what advanced
statistical methods are acceptable or reasonable? From the statistical, I
think there’s the data side and the stats side and
they go together about what’s allowed, what’s okay. – And I’d just add, one of the things I tried to frame up was I think the idea of reproducibility is key. You brought it up as well. I think, though, we must consider that you’re going to
need a data compendium, but we don’t want to lose
sight that one of the really neat things about real-world
data and translating to real-world evidence is that
we can save costs and time. And we don’t want to lose those elements to get the patients who
need the therapies quicker, we don’t want to lose that in the fact that we’re building and
building and building a compendium so we have
all the right elements to come forward, what is good enough, in terms of missing data? For example, the VA is
a tremendous database. It’s outstanding, if you
are particularly interested in heart failure, as we are,
and committed to this space and those patients, you
can get echo parameters, extremely deep data. The gender diversity isn’t
maybe what we would like, understandably, in that data set. So we need to look to other data sets, because you’re not going to get everything out of one source. But at some point, how
many studies do we build into the compendium and
where are the parameters to address the missing data
questions, the bias questions? – And then I guess the
last thing I would add is that what endpoints are relevant for individual therapeutic areas? Because you can build a great data set that is as good as good can be. You can start to say, okay, these are the statistical methods that
we feel comfortable with and that we can use to analyze this data. And this is the level of
advancement we can go to. But then we also then need to say, for each individual therapeutic area, what are the endpoints that are
relevant to make decisions?>But then also, hopefully for registrations or label enhancement purposes,
plus efficacy and safety. – Great. I would like to, I said for this session,
we want to have plenty of time for comments and
questions from all of you here and those of you who are joining us online around how we can support
real-world evidence development and accelerate effective regulatory uses. So I’d like to open this up for comments and questions at this point. So anyone, I see a hand up in the back, (man speaking off microphone) Okay, hang on one second, we’ll get a microphone to you so everybody can hear. – [Tim] Hello, I’m Tim Lance
with Century Data Systems. So I’ve heard I think
everyone on the stage in a couple other sessions
talk about the importance of reproducibility when you’re
using real-world evidence. And I wanted to ask the question, how do you see
reproducibility playing out, given the fact that data sets themselves are very inherently different? Can you actually develop a method that’s going to be reproducible
given the heterogeneity of the data sets that might be used? – I think it’s one of those answers that you’re going to hate,
because it all depends. I think it’s important to
look at different databases to ask the question. Then if you get different outcomes, then you look at it to try, that’s only the beginning. You do a study, you get results and then you have to understand them. So is there a reason? Is it because the Medicare
database has patients that look like this and the VA database is only men? And has a certain kind
of patient population. So I think if you do
get a different result, it adds to the richness of what you need to understand in interpreting the data. But I think those are important steps. Understanding the
diversity of the response and understanding the diversity
of the results, I mean, can really add to your understanding and adds to the richness
of the data that you get. – And what’s your sense of the state of the science in this area? It seems like kind of an
extension of meta-analysis and are we really at a
point where we understand how different, whether
it’s claims data properties or different other key
features that data do, predict or influence the
results that you get? – I’ll let others respond as well, but I think it’s an evolving science. I think we’re getting much
better with more experience and again, I think we have
a lot of great experience. Especially with the Sentinel experience, the ODYSSEY experience and others. And it’s something that
I’m sure will be evolving over the next couple of years. – Yeah, I agree on that the databases could be very different. But I think because they was
set up for different purposes. So now that we actually have a goal to use these data to support regulatory decision, or some support some sort of
healthcare decision-making. So if we have that common
goal, actually we can help shape the database
to somewhat more similar. They collect similar data and
use similar methodologies. But that definitely is
going to be a journey and require a lot of
collaborations across. – I would just add that I think there is a call to action to have more national fully linked data sets,
we have so very few. But we have such a diverse nation and forget about if you
want to have something that’s globally applicable,
that’s really difficult. But speaking from a national perspective, we have such diversity, we
must challenge ourselves to develop additional data
sets or this evolution will not be as rapid or as complete, that we’re all talking about, as we want. – Other questions or comments? Yeah, over here. – [Ritesh] I’m Ritesh Jain, EMD Serono. Since we are talking a
lot about collaboration, I just wanted to make a
comment about TransCelerate, which is a platform where
member companies across comes along and I think
they have an interesting work stream, which is
working on collecting placebo data from controlled trials across different therapeutic areas. I think that’s an interesting
work stream that’s going on. – TransCelerate has done
this pre-competitive work in a number of areas. And it does seem to fit
– Absolutely. – with some of the goals here. – Yeah, it’s certainly a model for working together pre-competitively. I would say there’s also a place for trusted third parties to
be a convener for situations like this and that’s
also extremely valuable. – Question, yeah? – Hi, thank you so much for the panel. And for taking my question. With respect to the third parties. – Oh please be sure to
tell us who you are, too. If you don’t mind.
– Oh, I’m Sonia Pulgar, I’m with Ipsen Pharma. I was wondering if you could comment on the oncology space,
for the role of cancer research networks and
maybe for us to collaborate with respect to sponsoring and supporting cancer research networks? There’s one I think that’s
quite active already. – I think that as an oncology person, I guess I can start. But yeah, absolutely,
and I think we’ve seen on the evolution we’ve
talked a little bit about when clinical trials, when
we see this hockey stick, I think the sharing of next
generation sequencing data, when we went from TCGA
to what we’re seeing now with GENIE and real broad
sharing of data there. And how quickly that that has gone, I think that’s been really helpful to see that happen in a positive way. So I do think yes, absolutely, that some of the bigger
cancer research network collaborations would be
definitely be a way to go and we’re very interested in that. – And I think especially
for platform trials in the future and I think
it’s not only going to be the cancer networks and the
physicians who drive this, but I think in the
future, patient networks and patient groups will drive this. And of course it’s in patients’ interest, people’s interest to think
more about platform trials rather than just setting one
individual product at a time. So I think that’s the future. – I think medical associations,
I would add, as well. That’s happening.
– Yeah, absolutely. – And there’s much more sophistication, but they’re really a fantastic partner to pull everybody together. It’s a really effective way.
– Yep. – Thank you. – [Juliana] Hi, I’m Juliana
from Deloitte Consulting. And I just wanted to add onto that, and the conversation that you had just now sort of leads into this. I was also thinking about
disease advocacy organizations and the role that they would play in terms of collecting and aggregating the data. And not just the biomarker
data that you’re talking about, but also a lot of these
patient recorded outcomes and the patient experiences both of being on the drug and also some
of the less quantifiable outcomes that we we’re talking about in terms of time spent with children and time spent being happy
instead of miserable. All of those kinds of
outcomes that so hard to get out of an EHR
and that are so critical to people’s lives and how
they feel about a drug. And just not readily available. But being able to bring that to scale in order to have the
statistical power that we need to be able to actually draw
some conclusions based on that. – Yeah, I think patient advocacy groups are growing in their
sophistication and ability to do even their own analytics, I think. But I think the beauty in
there comes with partnership. Because it depends on how
do want to you specialize? And they’re so incredible about especially coming with a patient voice
and bringing that forward. Do we really want them
to be the specialist and building the analytics and doing that? I don’t know that that’s the
fastest and best solution, but making sure the
partnerships are there. And again, from the
platform concept in terms of bringing people
together, I think that’s where you really want to bring them in. And again, have the patients at the table as a builder, from idea forward. – Hi, Jason Harris with the
National Health Council. I’m just building off of that. About a month or two ago,
we hosted a round table, just on that the patient
perspective on real-world evidence so just thinking about
the different skills and tools they would need in
order to really get engaged. So I’m just curious what
are your guys’ thoughts on how would that help
you in your industry, if you can some of
those views incorporated into what they need and they
can have those skills to then. – We’re looking into
that question and Jason, while you’ve still got the microphone, I know National Health
Council recently had a new report come out related to steps that patients can take, or
from patient perspective, would matter for real-world
evidence development. Do you mind expanding on
that a little bit, too? – Yeah, absolutely, thank you, Mark. Actually we just recently
released it today in alignment with this meeting. – Very recently (laughs).
– It’s a white paper on the patient perspective on
real-world evidence. So we went through just the basic concepts of let’s get a standard definition, similar to what we’ve heard today. And then thinking about
what do patient groups and individual patients need to know about real-world evidence? Whether it be some of the data concepts or the different terminology
that comes from it. And then starting to
think about the skillsets. So talking about registries
or what do you need to engage in trying to
develop things like that. But certainly, check out our website, happy to share it, it’s
about not too long. – Thank you, comments? – Yeah, I guess one idea I have is that maybe step back
from real-world data and just even think about
clinical trial data. Or clinical trials, how
can we actually follow up the patients after they are off the trial and then that could be
an opportunity for us to pilot some of these data
collection with the patients and partner with them to really understand what are the relevant outcomes? And importance in data
that are being collected after they off the trial
that can help inform their decision on the medicine. And also on the health studies. And that could be maybe an easier pilot also, for us to get on. – I think the patient advocacy networks, I think they’re incredibly powerful and their input is incredibly meaningful. And I think we’re just at the beginning of seeing the impact. I recently had the
experience working through the IOM, or in Nasum
on, as we talked about clinical trial data sharing. And they were thinking about,
from their perspective, if they give a grant to a researcher and then that researcher doesn’t share the clinical trial data
or the data, I should say, with other researchers who
are working towards the same goals of conquering a certain
disease that they have, then how do they feel about it? And when it came down to it, they’re thinking about, well, maybe they should have their own
guidelines on clinical trial data sharing because they’re giving money and they should set the bar. So thinking about that,
and this is what we’re all aiming towards in the
learning healthcare system is we should hear from
patient advocacy networks, from patients, from people,
what is important to them. And how we should approach
real-world evidence, real-world data, and then be able to try to meet those needs with
what we’re developing and working towards. – Thank you, back here and
sorry, before you start. Actually I know there’s another comment. Maybe just to help keep us
running ahead of the comments, hands up for just a second
for others who have questions? Okay, great, thanks. – [Michael] Michael
Liebman, IPQ Analytics. All of the discussion today
about real-world data, real-world evidence, a
lot of the operational or technology based issues, data sharing. I’m curious as to your reaction. Just yesterday there
was a paper published, or described in Medscape,
where pathologists had, a large bank of pathologists,
over 100 I believe, had about 1,500 specimens
that they measured. I believe it was a melanoma, over a period of six months with repeat measurements. And the concordance was about 70% for stage I, about 80% for stage V, but about 50% for stage II, III and IV. So the reproducibility among pathologists in that kind of study
indicates that the real data, real-world evidence
data, doesn’t necessarily have the same reproducibility
or confidence. And I wondered how you
were anticipating that. We did a study in breast patients where we used two FDA approved markers and saw some much similar effects measuring HER2 with IHC and FISH and finding that again, zeros and IIIs were fairly concordant, but Is and IIs have a very large disparity. And that data is not necessarily captured just in the definition in the fields that we’re talking about sharing. – Yeah, I’m happy to speak to that. Absolutely, as we have looked at biomarker data in the context of
real-world evidence, I think the confidence in the testing that’s being done, the
lab that it’s being done, the platform that it’s being done with. And data sets where you
have that level of detail are crucial and I don’t think we can be flip about how
important knowing that is, nor should we underestimate
the regulatory hurdles that understanding that data
would meet before we choose. And I do think to me this
is something that folks that don’t marinate in biomarker
work on a regular basis and think about companion diagnostics and registering drugs to test,
don’t necessarily appreciate. So when we start to
look at these data sets around oncology, the ones
that have high-quality biomarker data where we
can take the provenance of the marker all the way to the studs. And then as we talk to
regulators about that, I think that’s incredibly important and it’s something that’s
often glossed over. So I think it’s a really good point. – Yeah, seems like an issue
for real-world evidence, but also an issue for real-world care. – Yes, absolutely.
– Absolutely. – As well, yes. – Hi, this is Fred Yang, KBP BioSciences. So we mentioned about the real-world data in the specific therapeutic area. We mentioned the cancer, oncology. So what other therapeutic
areas that would have great use of real-world evidence, especially pre-approval
that could expedite the new drug for the patient? I did some antibiotics,
I know like pseudomonas or all of those infections are rare and really hard to find. And the regulatory pathway, obviously, you can not do the large
controlled clinical trial, because it’s really hard
to find those patients. So is that something the agency and all of the industry
leaders are thinking about? – Maybe I could start. I think diseases that
are largely symptomatic, where the regulatory
endpoints, FEV1 was brought up in the white paper,
but I’ll speak to PASI, which is a commonly used regulatory endpoint for psoriasis. A coverage of body surface area endpoint. The challenge is the PASI’s
not using the real world, and it’s really not
clear to understand from a patient perspective or
a provider perspective. So really understanding
the symptoms of, let’s say, a dermatologic disease are critical. And arguably more important even. In particular, when you
have a symptomatic disease like that, and more and more
diseases that are complex and have increasingly
complex symptomatology, I think it’s paramount. – Yeah, so I would add that
in the rare disease setting, when you have a hard time
even finding the patients, I think using real-world
data, looking at registries and also in the pediatrics
drug development setting, that would be another area that I think leveraging real-world data
rather than think about always doing large clinical trials would be another relevant setting. One more would be in
the neuroscience space and this is an area that
depending on the disease that we are talking
about, but for example, in Alzheimer’s disease where
this a long progressing disease and we are not going
to be able to follow up the patients in a clinical
trial for 10, 20 years. So do we again leverage real-world data to really understand
the disease progression and identify treatments that
can benefit the patients would be another area that
we could use real-world data. – Thank you. – [Bill] Bill Crown, OptumLabs. Just a comment about research design as opposed to methods. I think we tend to focus on the methods. There’s been so much methods development. But actually, my own
personal opinion is that the methods are in a pretty good spot now. And particularly with all of the work that’s been done in the
safety area with Sentinel. With the quasi-experimental
design, we’ve gotten to a point where the method
are pretty well established. And the studies really
don’t vary all that much with different methods. What’s much more important
is the study design and how you frame the question. So Women’s Health Initiative
is a classic example of this. And Miguel Hernan and
Jamie Robins at Harvard developed this idea that they, a concept called the target trial. And it’s the idea of designing
your observational study like it was a clinical trial. And my sense is that what we’ve learned from the safety literature so far is that when you pick new initiators
on drugs, for example, and you have a thoughtful comparator that you’re paring with the intervention treatment that you come to
something that is more reliable, in terms of the evidence
than if you were to just take all comers or do a less thoughtful comparison. So I just want to put
in the plug for thinking in a regulatory environment
about what’s the guidance in terms of research design? Because I think if anything, that’s even more important than methods. – I see nodding. – I think we all fully agree.
– Absolutely. – One of the things that’s being piloted in ODYSSEY is that the design,
the protocol, let’s say, for studies is done
collectively by the group of interested parties and
people who want to participate. So you get that richness and I think Marc mentioned that earlier. You get that richness of
input before you actually do the study, rather than waiting ’til it’s criticized afterwards,
when it’s published. Get as much of that rigor ahead of time and thank you so much, that’s
a very important point. – Yeah, so here, here and then Bob and that’s probably going
to close us out, go ahead. – [Audience Member] (mumbles)
so I just want to comment. So as an assumption that commonly may be not only part of the product. It would also be part of a
service beyond the product. Patients of only programs
that they resemble. And the patients of any
program that’s happening in the real world. So it’s very challenge to comment by communal trial settings. And it’s very important for us to generate the evidence on the
service beyond the product. And then also the critical job to communicate this kind of evidence to our stakeholders, patients,
payer, regulatory agencies. So it would be great if we
can have some clarification on how we can generate the evidence, how we communicated that kind of evidence. Even potentially
incorporating the labeling, so it will give us more flexibility communicating this kind of evidence. I think that’s critical. – Both on label and it
sounds like you’re talking about maybe some off-label
communication, too. Comments, yeah, okay, agree. Thanks, Marc? – Marc Berger, so in prior
recommendations from ISPOR, we had task forces on
methodology about how to do appropriate
analysis and comprehensive analysis of observational data. One of our recommendations, which hasn’t been paid a lot of attention to, and I think might be of
particular interest to the FDA is that when you go into
do an observational study, and that the insight is
in a different population or it’s based off an insight of an RCT, to do a first analysis where you restrict the analysis to a
population that looks like the population that was in the RCT, to see whether you reproduce the finding. If you reproduce the
finding and then you find something else, you would
have greater confidence that what you found makes sense. If you don’t find the same answer, then you have an opportunity
to ask the question, what was different here compared
to what went into the RCT? So it’s not even just about doing an analysis off a data set. It’s actually understanding
the insights in that data set. And I think that Bill
is absolutely correct. Design is paramount. And I think we need to
find a way to bridge that anchor between the
RCT world and in the observational research world. – Again nodding, other ideas?
– Emphatic agree (laughs). – Yeah, absolutely.
– Other ideas on bridging RCT to real-world evidence? – So I agree very much with your point. I think it really help
build that confidence with the real-world data. I guess the other way that
I would look at the bridging is the reverse bridging
of what are the endpoints and data that are relevant
in the real-world setting? And knowing that, for
example, in oncology, resists is not being captured or used in the real-world setting
or clinical decision. So how can we bridge it
back to clinical trials? So then the data that are
relevant in the real world for treatment decision
and clinical decision would be captured in the trial. So then we have another
way to bridge back. – Great, great comments. And Bob, I think you’re
going to get the last question, comment for this round. – [Bob] Actually, I wanted to
ask about Alzheimer’s disease. I presume the area of interest would be in something that delays its progression, or prevents its occurrence. It couldn’t be just
symptomatic improvement because we all know those effects are too small to be picked up this way. That seems a potentially promising area, if one could define very well who the at-risk population is. Presumably, before they’re overtly sick. It’s not an easy thing to do. But you can imagine endpoints
that would be credible. But they probably are
two and three years away, so it sounds like a tough area, unless you’ve got a lot of time to wait. – Endpoints, I think at
least two or three years. – [Bob] Endpoints would
be an early diagnosis of mental dysfunction. Or a next stage or early
or something like that. But all of those could
take months to years. Six months and a year. But it seems sort promising, because those are things that are
picked up in the system. You know when someone
has mental dysfunction. – Is there a way to keep the? That’s going to have a
longitudinal component, I think. – Absolutely, yep.
– And is there a way to keep the, this one,
Alzheimer’s did not make your early list of use case opportunities. – We’ve certainly put a lot of interest in it.
– It certainly is an important one, so yeah, let’s talk about that for a minute. – Yeah, I think to add a
retrospective component when you have that is so important. That is incredibly valuable. I agree you get a longitudinal, but you can learn so much retrospectively. And that is a full continuum. – [Bob] That could also
help you design a proper, a regular study, not a proper
study, forgive me, sorry. We don’t know that much
about the natural history of people with early disease. Maybe you could find out
if people progress faster. – Yeah, I think it requires
us to be able to go back to look at the data and
they was saying the data that have been in the system may not be adequate to look at that. But certainly, I think this is an area that has a lot of potential and promise and would be of really critical
for us to get that one. – I think when you were talking
about a national commitment, that’s where linking data
sets, where you would have, kind of where the patients
being diagnosed with Alzheimer’s and you’re able to go back
into a course of history that is longitudinal and
then digital biomarkers and a lot of the things
we’ve been talking about with wearables and other things. And what’s relevant to
patients and all of that going back in time, I think
one of the big challenges is where we’re seeing
two and three years ago, the stuff you’re talking
about isn’t necessarily linked to where they’re being diagnosed. And I think that’s where that national commitment really comes in. – [Bob] That really says that this hasn’t been talked about that much. One of the things to do with this stuff is to get better natural
history data than we now have. Potentially as a control group, but also just to help
you design the studies. – Yeah.
– Absolutely. – Well, I want to thank the panel. You all covered a wide range of issues in getting to, as we
talked about, a much better and reliable evidence and a wide range of applications in getting there. Ending up with Alzheimer’s seems like a really appropriate place
to finish this panel. So thank you all very much.
– Thank you. (audience clapping) – We’re going to take a 15 minute break. And our last major session
starts at 2:45, thank you. (people chatting distantly) Final session for the day. I want to thank everybody
who’s stuck with us for the whole day, been a
lot of really outstanding discussion and in getting
ready for this last panel which is charting a path forward, talking with Jonathan, he said, “Okay, well now it’s time for
people to start doing stuff.” So that’s what we’re going
to talk about in this panel. We want to discuss some of
the ideas that came up today and also talk about some
activities that are underway now, related to real-world evidence. That will give us a chance to reflect on the discussion of the day. To maybe think a little
bit more about priorities. And especially, Jonathan,
practical steps forward that we can take from here. And very importantly, we want to make sure that we’ve got a central patient focus in the work that we’re doing now. So that’s going to be an important part of this panel as well. So with that, I’d like to
introduce our panelists. At the far end, Greg Daniel, who you heard from earlier. Deputy Director here at our Duke-Margolis Center for Health Policy. Next to Greg is Jonathan
Jarow, who’s Senior Medical Director at the Food
and Drug Administration. Who has been involved and
done a lot of thoughtful work related to real-world evidence. You may have seen his paper
on this topic recently. Joe Selby, is the Executive Director at the Patient-Centered
Outcomes Research Institute. And Preston Hinkle is a
member of the Cystic Fibrosis Foundation’s Patient and Family
Research Advisory Committee. Preston, we’re really glad to have you here with us today, too. So we’re going to do the panel in the same kind of structure that we had for our earlier panels,
with some additional opening comments, thoughts from
the group about some of the activities that they have
underway already in these areas. And feel free to add in or reflect on or push from what you’ve heard about during the course of the day today. And if you don’t do it, I will,
as we get to the discussion. So Greg, please go ahead. – Okay, thanks, Mark. So I started out with some talking points this morning for today’s session. And I’ve completely abandoned them and throughout the day, I’ve been jotting things down that I’ve
head that gave me ideas. So bear with me as I
read from chicken scratch and I’m trying to figure out
exactly what I wrote here. But, in terms of high-priority topics that I heard either today or things that we outlined in our
white paper as things that would be very helpful
to start doing now. Some of these things we’re already doing, but they need to be
coordinated, are as follows. One, improving standards
in data collection methods. So we know real-world data
sources are very disparate. They cover a wide range of types of data and different sources
and there are challenges in making sure that when
we’re using real-world data sources that we’re actually
measuring what it is that we think that we’re measuring, and that we can interpret the data. So a lot more need to go into standards and better data collection. Another on is strengthening
the methods for randomization in the clinical setting as well improving the credibility of observational studies. And we heard a number of ways to do that, but more does need to come together. And how we know that the
observational study is credible? And how can we best do randomization in the clinical setting? What I also heard today
is this idea of doing more common platform type studies. Thinks that like PCORnet could support or other networks like that, where you can get a range of companies
or a range of stakeholders interested in doing really
high-quality real-world evidence generation for
a number of products. I also heard, and this
is something I made up on my own, but I was
listening and I’m forcing it in here is basically
what I’m trying to say here. Evidence-based approaches for matching what we were talking about earlier, when we talked about research design. And Bill, you were talking a lot about, it’s not necessarily the methods, but it’s the design of the research itself as a gap and linking clinical trials to real-world evidence generation. But the bridge that I’m creating now is then how do you
match up research design and all of the different flavors of that within real-world evidence with
actual regulatory decisions? Simply, which types of
regulatory decisions can be very well done
observational studies support? Which types of regulatory decisions would you absolutely have to have randomization. But as you get into the details of that, it’s not as simple as that. There are a big range of things. But we do need stakeholders
to come together and really put a lot of thought into how you make those matches. I did hear today engaging
patients in a better way to learn about real-world
evidence and real-world data. But also to engage patients
and have them participate with their data, but also
participate in research design and communication of results. Governance was a big thing
we heard this morning. That will continue to
be an important emphasis to look at, both public
and private sector ways to share data better and
to build the infrastructure that we need to do more of the
common platform type things. And then novel analytic
techniques that take better advantage of
today’s computing power. So every day, or maybe every week, maybe, brand new methods or more
sophisticated methods for how to do real-world
evidence development are out there and being tested and being further refined and developed. We need a good place
to continue to do that. So all of these things can be done with demonstration projects. But demonstration projects are great, but we do need them to be
coordinated and transparent. It’s unbelievable how
many different groups and partnerships today that are all trying to do something in the area of improving real-world evidence development. It’s going to be really hard
to know exactly who’s doing what, what kind of projects
are currently happening now, so that we can learn from those. Understand where the gaps continue to be, is everybody focused on
replicating clinical trials? Is nobody focused on
some of the other things that we talked about? So it will take collaboration
and coordination. There are a couple of things coming up over the next year that I think are going to make a dent in some
of that collaboration coordination I just talked about. One of the things is
the National Academies, their forum on drug discovery,
development and translation. They’re about ready to launch a three-part workshop series on how to do
real-world evidence better, with a focus on methods and
approaches across a range of medical technologies,
not necessarily drugs. But will be focused on
methodologic approaches to better informed real-world
evidence development. And parallel to all of that, our center, the Duke-Margolis
Center, will be launching a real-world evidence
collaborative in October. Where we’ve brought together
a range of perspectives. And we’ll be looking at supporting and broadening existing efforts. So some of the things
that Bill talked about in terms of what Optum is doing. HealthCore is doing a lot, as we heard from Kevin earlier this morning. And a lot of other groups are doing things that can be impactful
and will be impactful. And we’re looking within our collaborative to help learn from those lessons, to help identify where
groups aren’t working and then to build on those. So we’ll be conducting
a series of workshops. We’ll be producing a series of papers and identifying demonstration projects that can actually help
fill in the gaps of where some folks aren’t necessarily
working or focused on. I think that’s everything on
my chicken scratch I can read. – Thanks, Jonathan.
– Well, I, too, did chicken scratch during the day. So first of all, I would like to thanks the folks at Duke-Margolis
for hosting this event and for putting on a wonderful meeting. In addition, I’d like to
thank all the panelists for the excellent
conversations we had today. And the people from the audience
who participated, as well. FDA is looking for stakeholder
input on this subject. It’s very important to us
to hear from all of you in terms of where there are knowledge gaps that we need to help
fulfill, either through pilot projects, guidance
development, et cetera. So real-world evidence is very exciting. It’s a very sexy term, I’m surprised it’s not on the cover of Science, or maybe it’s not really
science, so (laughs) it might not make it there. But at any rate, it’s very exciting. And I think the learning healthcare system is something that we all aspire towards. And I think various
regions are ahead of us, in the US, in getting the interoperability and standardized data sets and stuff. Sweden is someone that
we aspire to achieve something like that or what happened in the Southwark Project is a great thing. Because we all feel like we’re
losing a lot of information. And patients are on board with this. I remember at a Friends of
Cancer research meeting, Mark, I think you co-sponsored
it or helped it along. Patients were aghast that the information about their treatment wasn’t being used to help guide treatment
for future patients. We heard numbers today of three to 5% of cancer patients are
enrolled in clinical trials. So there’s a great deal
of information lost and we hope that we can
capture this information and use it, both for regulatory purposes, but also, there’s probably
even a bigger interest amongst payers to make decisions about whether to pay for drugs and
how much to pay for drugs. I know in certain regions of the world, they again, market access initially, but are required to
provide real-world evidence of the effects of the drug
after it’s on the market in order to continue to get reimbursement. And in addition, obviously
clinical best practices would greatly gain from having all this information brought to bear. Most of our best practices
are based on expert opinion and not based on level one evidence. Again, whether the
real-world evidence through observational studies would be considered level one is another question. So I want to say that I
hope it’s demonstrated today and at other events
and in our writings and speakings that FDA
is in favor of this. We don’t want to be viewed as a roadblock to utilization of real-world evidence in the regulatory setting. And in fact, as I think
has already been mentioned several times today, we’ve
been doing this for a while. Sentinel has been used
for safety, primarily. And we’ve approved a variety of drugs where demonstration of efficacy, I wouldn’t call it necessarily
real-world evidence, but certainly based on
observational findings and certainly with single-arm
trials where we use external controls, which people
call historical controls. There is a great deal
of experience already at FDA in doing this so I don’t think it should come as surprise to you. And we already have companies
that have approached us to discuss potential
projects as well as a recent or soon to be approved
drug where they used real-world evidence as
I think one of the other panelists mentioned,
one of the oncologists, to show that that population
was not prognostically different from other sub
groups of the same disease. So I think that there’s certainly a role for real-world evidence. We encourage people to pursue this. I think that interactions with us, either through feedback on the white paper that was distributed
prior to this meeting, or contacting the Office of Medical Policy through the mailbox that they have there with questions or complaints or ideas. Probably no complaints, right? And in addition, meeting with FDA as you’re considering,
at least industry folks, as you’re considering ways to incorporate real-world data and real-world evidence into the development program. And as has been mentioned
repetitively today, that could be used to just
understand the disease better, used as historical controls,
used in combination with a traditional clinical trial. Or in place of traditional clinical trial. I think it’s very clear
that there is a lot of room for development and
improvement in this area. And I put it into three buckets. The first one is data,
and this has been said over and over again, but that’s
the bucket of deplorables. Is the data there? Is it reliable? Do we have data standards? Do we have a unique
patient identifier to get? – Can somebody tweet that?
(audience laughing) – Cross sectional and longitudinal. And that’s one of the
problems we have in the US. Unlike Sweden, we do not have
unique patient identifier. Which would help, but if
we were using social media or any of these things,
pharmacy, pathology, how we do know that the, linking the patient to the
drug that they’re receiving is very difficult in our current system. But enough said about data. The second bucket is study design. We had some comments recently about the importance of study design. I think I’m glad to hear today that people repetitively said that real-world data and real-world evidence doesn’t mean you have to exclude randomization. Obviously, FDA is very enthusiastic about randomization within
the healthcare system. There are other names for this, large simple trials, pragmatic trials. I am disappointed and I
think the onus is on us, that at every one of these meetings, people say, boy, I wish
FDA would accept pragmatic clinical trials for regulatory action. And Bob Temple and I sit there and go, “Why are they saying that? “Of course we would accept it.” We obviously have not been good at getting that message out. If you do a randomized controlled trial, it’s a randomized controlled
trial and we’ll look at it. And this brings up just a
little tangential point. Definitions are important,
we spent time defining what we call real-world data, what we call real-world evidence. The bottom line is, whether
you call it real-world evidence or not, FDA is going to
review an application for a demonstration of efficacy based on the statutory requirement of establishing substantial evidence of effectiveness. And we’ve historically used
a P value of less than 0.05 in adequate, well-controlled trials. Sometimes just one trial. But that hasn’t changed. Not the P value part, but the
substantial evidence standard. Actually the statute and the regulations do not mention P values. So there are alternative approaches. And we live in a very exciting time for regulatory science in that in addition to real-world evidence, which
is in 21st Century Cures and FDARA, there is
also this codicil about doing innovate trial design and looking at simulations and Bayesian
approaches, sequential analysis. So there’s a lot going on, both here, in the United States as well as in Europe. And I know that the adaptive pathways, adaptive licensing pilot
which is now closed, but they’re now working ADAPT SMART. And they’re looking at incorporation of real-world evidence in the development of drugs for EMA regulatory actions. So I think there’s a lot going on there. But having said that, back to
the bucket of study design. There’s randomization and
then not randomization. I think everyone’s got it in their head that this is really all
about, even though we want randomization, I’m not
trying to discourage that. All these discussions
are really about doing a study design that doesn’t
include assignment of treatment. And that’s where you run into trouble. Because establishing causal inference without randomization can be difficult. And everyone says it’s
based on effect size, but basically, it’s
based on our confidence that whatever confounders,
both known and unknown, that is exist do not wipe
out the effect observed. That’s really the bottom line. Marc and ISPOR has done a wonderful job at trying to come up with a set of ways to do this that would
create more confidence. But again, the FDA has
historically approved many drugs based on
evidence that was anecdotal. I think the smallest approval
is Cholbam with 12 patients. So we do have regulatory flexibility when it’s clear that the substantial evidence standard has been met. The third aspect really falls in our lap, and that’s regulatory. I think a lot of people need reassurance that good clinical
practices can be followed. And that FDA will accept data
in review for the application. I think that’s one of the
uncertainties that companies have about embarking on this area. Our current guidance on this, ICH E6 R2, really doesn’t deal with
these non-traditional trials. And so it’s incumbent on
us to revise that guidance. And that work is already
happening to do that, but I would not expect
that within the timeframe, of the framework that we have
here for real-world evidence. So I want to just, some final points. FDA continues to support Sentinel. It is exploring ways to use that for things beyond just safety analysis. Again, we’re open to feedback
on all of what happened today, the white paper, other
issues that come up, and that can be either
done by sending something to the mailbox or a meeting with us. Thank you.
– Thanks, Jonathan. Just to be clear, that mailbox is not like the mailbox outside of the FDA office. – Right.
– It’s an online. – Silver Spring, yeah.
– It’s an online place. – It’s CDEROMP, one word, – [Mark] It was on
Jacqueline’s slide earlier. – Okay, it’s on her slide. – Joe?
– Okay, thanks, Mark. And Jonathan, there was one group that you left off your thank you list and maybe you felt you
couldn’t, but I think the FDA should be thanked very heartily. It was you that called
for the meeting to happen. It’s your presence here,
I think, in large numbers that contributes to the
rest of us being here and to the energy in the room. So we’re very appreciative of
you having convened us all. I want to start with a little
comment about progress. I got here about six years ago as PCORI was getting off the ground. And the first meetings I went
to about real-world evidence were more along the lines of, is there any possible use for it or no? Is the fact that doctors
are entering this data and they got it wrong the last time I went to the doctor’s office and that proves that it’s worth nothing for any purpose. We’re not hearing that anymore. And we’re actually now
hearing just an amazing array from a lot of
different corners of interest from natural history, the
use of it for natural history studies and improvement
and enhancement on natural history gathered in many other ways, to predictive modeling, which we at PCORI are very excited about,
to machine learning. So all of those I think are, we have thought about them at PCORI. We’ve thought about them at PCORnet and appreciate that
really, there are aspects of precision medicine, there are aspects of getting the right
treatment to the right person. So very exciting to see that
so many people in this room have an angle on that that
points in the same direction. So we talked about, I
think this is next steps or the way forward or something like that. So I have five things I’d like
to just very briefly mention. I have to say a few more words on linkage. Not that it hasn’t been
talked about a lot already. Talk about societal change, something that Kevin brought up and I mentioned earlier. The power of demonstration projects. Standardization, trial infrastructure. Those are the topics. So first, with respect to linkage. It hasn’t been quite said today that you really can’t do anything in observational analyses
if you don’t have a denominator like the denominator
provided by claims data, whether it’s a commercial
plan, Medicaid or Medicare. So all the data that EMRs gather is on an unspecified cohort of people. You don’t know how many
people would have used that system, but just didn’t. You don’t know how many
people used that system for part of their care,
but had a diagnosis or an outcome elsewhere. So we think that for either
observational studies or really for efficient large trials, linkage is impossible and
PCORnet has that challenge because we started with EMR
data and learned over time how tough the linkage issue is. We’re currently co-funding with the FDA several small projects on linkage, both in devices and in drugs to be. And particularly looking
at possible linkages between Sentinel, a claims database with well over 100 million Americans in it and PCORnet, an EMR based system with well over 100 million Americans it. Probably not the same 100
million, but substantial overlap. It’s very interesting and
just a last word on linkage I think, is that the
thinking behind PCORnet is it isn’t going to work
is the holders of the data, from the patients to the clinicians to the systems to the payers,
if they aren’t involved. And it’s quite interesting
that the linkage problems are really between the delivery
systems and the payers. Now that’s an interesting pair, the delivery systems and the payers. As somebody said, they do
business together this morning and that’s a challenge
that’s part of the barrier. But can you imagine the
relevance of the questions to which the systems and the payers agreed this is a relevant question? This is a question worth
contributing our data and sharing it with the
other side, so to speak. So this notion of bringing
maybe unlikely partners together around data is just critical. It’s very interesting to me
that it’s come down to that. Connected to the question
of linkage is this question of society and particularly
the role of patients. And today has moved me, I think. I have long subscribed to the
notion that it’s unethical not to do the most you can with the data that are being generated
in delivery of care, to learn how to do that better. Ruth Faden and Nancy Kass’s
argument from several years ago, I’ve always subscribed to that. But I think, and I took note
that Jason from National Health Council said that they had just convened a group to talk about real-world evidence. I took note of what Sally and Kevin and others said in that first panel today. And I think I can say
that as one way forward, we would like to convene or co-convene or join in a serious
meeting on this question is it time for patients,
and the community, patients, people, to
begin becoming more aware of the fact that these
barriers to linkage, barriers to the use of
data are getting in the way of improving care and to
have their voices heard. I think we need to have IRBs there. We need to have institutions
there, even more. I honestly don’t think it’s primarily a HIPAA or IRB problem. So linkage, so societal
change, this ultimately does lead to the learning health system, I just want to be on record as saying that I think a learning health
system naturally leads to the prospect of
hosting everything up to and including pragmatic clinical
trials with randomization. I think that once you get
into it, you’re stuck, you begin understanding
that research has value. And then you begin understanding that some research requires randomization
and there you have it. So I don’t think there’s a disconnect. The power of demonstration projects. So Rich has left the room, Rich Platt, so I can say that most of anything I know about big data I learned from Rich. And one of the things he
always told us on PCORnet and before that with
Sentinel, was that you learn most by getting started
doing the research. So you don’t actually
learn it by sitting down in a group and trying to
imagine all the problems you’re going to run into. You can find them in a
heartbeat by starting a study. So in PCORnet, we have several
demonstration projects. One is a very large study
of bariatric surgery and this joins data from
across 60-some-thousand, I think it is, maybe 100,000. But I believe it’s in the 60,000
range of bariatric surgery, including very large
numbers even in adolescence. So out of that, the researchers came up with a remarkable document
that was a series of learnings. And they were about the issues you expect about the degree of missingness
and about the importance of the degree of missingness. And the need to standardize
data across systems and how they’re not. So I truly think that ADAPTABLE
is the trial that Adrian Hernandez talked about a
little bit earlier today. And in ADAPTABLE, we’re learning about, among other things, e-recruiting. So most of the patients in ADAPTABLE have been recruited through the portal of their health plan, wherever that is. So we’re going to continue
funding demonstration projects. We have number of them and as I said, a couple of them in
collaboration with the FDA. But we’d love to see others join us, so we really invite you to get in there and projects that have
a piece of research, demonstration projects have a
legitimate research question, but you realize that along the way, you’re going to be blazing new trails. You’re going to be standardizing new data. That takes me to standardization of data. And we are supportive and we
fund data standardization. The common data model that
either OMOP or Sentinel or PCORnet have, any of those, is still relatively close
to what it looked like when there was just claims data. It started with claims data. And we’re only very timidly,
because it’s so complicated, going into the world of EMR data. We need to bring people together, clinicians and researchers
and holders of data. To talk about measures that are not in the common data model yet. Pathophysiologic measures
like lung function and heart function, ejection fraction. Just to name a couple. More sophisticated, more
complex laboratory measures, genetic information on tumors
and journaling as well. Data standardization
is an area that I think both takes the real-world
data to a new level to where it’s really useful. But it also spawns
research in the process. I have a meeting coming
up with the Oncology Center of Excellence
with Amy from Flatiron with ASCO and a number of others to begin talking cancer by cancer,
about what are the data elements in real-world
data that you really need to be able to do either
large observational studies, say you have off-label prescribing, or pragmatic trials, outcomes trials. So I just encourage
that illness by illness. And ultimately, I just ask
the question of whether you don’t agree that ultimately,
the data that we need for pre-approval trial should be the data and the outcomes we need
for post-approval trial ought to be the data that clinicians and patients need for clinical care. I think, ultimately, if you
want to be patient-centric, that’s the kind of outcomes
that you would need to standardize and include. Trial infrastructure, this
is I think the very last one. ADAPTABLE taught us that it’s one thing to want to do a clinical
trial in a real-world setting, you know, those of you
who do regulatory trials know that you have a vast
infrastructure to do them. And we are trying to conduct
trials that don’t intrude, or hardly intrude, on clinical care. But we don’t really truly
have the infrastructure there. And to do trials of serious questions like does changing a medication
make a difference? We are going to have to
invent a trial infrastructure. PCORnet and the Patient
Center Research Foundation, which is now the not-for-profit
that PCORnet stood up to help it with sustainability,
will be working on this. Will be working on beginning to strengthen the trial infrastructure
in real-world settings for pragmatic trials. So those are some areas
that PCORI and PCORnet will be involved in coming
up and with that, I’ll stop. – [Mark] Great thought
and Preston, really glad that you’re with us this afternoon, too. – Thanks, well, good afternoon. I am here today as a
patient representative. There’s been a lot of conversation today about the patient’s perspective and I get to be that voice, no pressure. I was diagnosed with cystic
fibrosis at the age of three. For those who don’t know,
CF is a genetic disorder that affects the regulation
of ions and water in the body. It causes bodily secretions like mucus to be very thick and sticky. That has a host of complications,
including in the lungs where particles, bacteria, can
get trapped, cause infection. Having a chronic illness like CF is incredibly formative. A patient has a perspective
that is really unique, from that of a researcher or a doctor. A researcher might approach a new treatment as what’s the data? What does the data show? A doctor would say, what are going to be the outcomes for my patients? As a patient myself, I
care about these things, but I have to think of
other considerations, such as, how does this
treatment make me feel? Or even, do I have time in the
day to add another treatment? It’s really going to
be important, I think, going forward with the real-world data, real-world evidence conversation to include the patient perspective. To bring that voice in. The Cystic Fibrosis Foundation has been an incredible pioneer in this regard. I’d like to highlight a
couple of their initiatives. For over 25 years, the CF Foundation has run their patient registry. This covers almost 30,000 individuals that which consists of about 84% of the CF patients in the United States. They collect data at every
visit, including health metrics, demographic information,
bacterial cultures. It’s a huge list of data
and it’s just a really incredible wealth of information. There’s so much there
that the CF Foundation literally did not know where to start. The solution that they
came up to that with I think was really incredible. They said, let’s turn
this over to the patients. To the caregivers, to the people that research affects the most. They put together a team
called The Patient and Family Research Advisory Committee,
which I was able to serve on. It consisted of patient
and patient family members. And we helped to launch
the Insight CF Project. This was focused on collecting questions from patients, family
members, healthcare providers, anyone with a personal stake
in cystic fibrosis research. And simply saying, what
are your questions? What can we help to answer? Here’s the information that
we have with the registry, what of this do you
want to know more about? So we had a really incredible response, over 400 questions submitted. Of those, about 150 were feasible. We were able to narrow that
down, resubmit it to voting and came out with three amazing questions that were sent to a research team. And the rest were held
onto, because they did give a really incredible
insight into the current state of CF patients, their
minds, their concerns. So this, as a patient myself, who was able to submit a question, was intensely gratifying. The ability to submit my own concerns and know that they were being listened to. It was just, it was a unique
experience, it was incredible. So I think that this
should be a role model for other organizations moving forward. Connecting directly with your patients. Listening to what they
want, what they need. Opening up the discussion to them. One other initiative that
I wanted to highlight of cystic fibrosis in the
discussion of real-world evidence is the label expansion in this past summer of a drug called Kalydeco. This is from a new class of
drugs called CFTR modulators, which actually go in
and assist the protein that’s malfunctioning in the
cells to achieve some function. It’s baby steps toward a
cure, it’s really shown incredible results, but
it’s limited by genotype. So the populations are
difficult to do traditional randomized clinical trials with. Over the summer, the
label expansion was able to be done using in vitro
data, which was backed up with a safety profile that
was garnered from sources like the patient registry
and other sources of data. And so it was, I can’t speak to the methodology, I’m
not an expert there. But as a patient, this
was phenomenal news. I personally am not eligible
for one of those drugs yet, so to me, as a personal
stake, expanding those fields, expanding the breadth
that these drugs can cover should be one of the most
primary goals of CF care in the near future. This is yet another example
of how real-world evidence can really make a difference
in the lives of patients. To sum up, I just would
like to encourage everyone, keep the patient’s perspective in mind. A patient is an expert
in their own experience. Use that, utilize that, and I think that by doing so it’s going to drive some really incredible results going forward. – All right, thank you very much, Preston. I guess the one thing, the one reason we thought it was very important
to have your perspective on this panel is, I know
you’re not a methodologist, but as many of you’ve heard today, the issues that stand in
the way of real progress on real-world evidence are not, they certainly involve methodology, but they also involve more basic things. Like going back to Joe’s list and some of the points that you all raised around patient support for the research. Around being able to
capture longitudinal data. Around being able to capture
meaningful information on outcomes or questions of interest. And then formulate
those questions in a way that everybody can get behind. Payers that might contribute data, researchers, clinicians
and of course the patients. It seems like a lot of the issues that we’ve raised here today
have been addressed in the CF platform that
you’ve been an intrinsic part of developing, not to
mention contributing to. And you did talk about
some of the implications and hopefully lessons for other areas. And we talked about other
potential applications today, too. Anything else to add on this point? I’d like to ask the rest of the group about how a model like
this could be applied, or what are the barriers to applying it in these other areas,
where we’ve got lots of use case examples that we talked about today. Where there is some level
of patient engagement, but not maybe to the
extent that we’ve seen with the CF work that you’ve described. So first, any additional
comments you’d like to make on this topic and then
I’d like to ask the rest of the panel about how
much of a model can this be for real-world evidence progress
in other use case areas? – Sure, so one other thing
I can highlight, I guess, is specifically in the
context of real-world data for real-world experience,
is the questions that we received through that
project that I was a part of were incredibly varied and
I was just blown away by the amount that were actually
answerable with that registry. And to answer some of these questions, using a traditional
randomized clinical trial would be a significant challenge,
just out of sheer numbers. However, with initiatives–
– So many of these genetic subtypes, there are
just a few patients, even though you found most of them
in this effort, yeah. – Sure, and so, in that registry, it’s really incredible because
almost 30,000 patients. That data is there,
that data, we don’t have to recruit people, to design a study for, we can just go to that and look. And the implications of that
to me are incredibly profound. – Greg and then I know Joe
has got something to say. – Just one piece of that
that I want to comment on is that Preston seems pretty motivated to participate in that data,
just by listening to you, I could tell that knowing that your voice was being listened to or
that something positive was happening with your data was enough to motivate you to
continue to participate. When we talk about
real-world evidence, we often talk about claims and
electronic health record data. And while there is
missingness within that data, it’s pretty much there. If you’re a provider or hospital, you need to submit the claim to get paid. So that’s going to be there. If you’re a provider that use
an electronic health record, by and large, you’re documenting data into that health record. I don’t know if it’s accurate,
we don’t really know, nobody really knows, but it’s there. So now we’re talking about unique data sources from patients. And there are ample ways
that, in technologies that we have where patients
can contribute data through the registries
like the CF Foundation or even getting into mHealth technologies and apps and things like that. But I think we need to
think about that fundamental element that I picked up on from Preston, is that it will depend on how
motivated the patients are to continue to use those technologies, or continue to, they don’t
need to do that to get paid like providers when
they’re submitting claims. And they don’t need to
document it to treat patients like providers and payers
all do those things. What is going to motivate
patients to contribute to those studies, to keep
using those wearable apps that are contributing to research? And something simple as
letting those patients know or having them experience
the fact that their data are being used in a
meaningful way can go a long way. And I just would encourage
groups that are looking into how can we better use patient datas to think about how contributing data can actually have an
impact on those individual patients that are participating. – So I just want to say first of all, it’s not a coincidence or just a drawing a ticket out of a hat, that
Preston wound up here today. That everybody thinks of the
Cystic Fibrosis Foundation as the paradigm and some happy coincidence of very broad coverage,
there was one really national cystic fibrosis patient organization, one. Number two, to my
knowledge, you really linked the foundation to drug development efforts in a very tight way and
I think, number three, you were a rare condition
with no good treatments, and so you’re highly motivated. So a lot of things happened right, plus it must have been just
unprecedented leadership, just incredible leadership,
but there are a lot of patient groups and we know a lot. I’m sure the Patient Powered
Research Networks in PCORnet who aspire to be like your foundation. So I think there could
be ways that we learned. – [Mark] Are you seeing
some promising developments in that direction? I know much of the PCORI
work has helped support. – We’ve just issued an announcement that actually is directed
only to the PPRNs. So the big CDRNs can’t apply for it. – [Mark] Maybe just back
up to explain the PPRNs. – Okay, good, sorry. Sometimes I forget. So PCORI funded PCORnet, which
is a very large investment. And it’s a network of
networks and in PCORnet, there are 13 very large
clinical data research networks, which are EMR driven. Every one of them has
several millions of patients. You add the 13 up and they’ve
got somewhere over 100 million persons with records in the database. So that’s the CDRNs, and then
there are the Patient Powered Research Networks and
these are groups often with an advocacy organization involved. Sometimes with an academic base but, of patients with a single
or group of very closely related conditions like
vasculitides, for example. Epilepsies for another. And they aspire to become
more involved in research. Some of them already
are, some of them have partnerships with
industry sponsors already. Maybe especially the rare disease ones. But we’ve just put out an announcement that will bring the PPRNs
together with another funder, they must bring another
funder, so industry sponsors, or other foundation sponsors. Advocacy groups could
join in funding of it. So we are doing what we can to try to make these groups, help
these groups get to the point where they’re more familiar
with, more comfortable with participating in trials research. And they’re very good at collecting data. Transferring that into
analyses that lead to cures, or I’m very interested in the questions you’re putting in your pocket. You know, the ones you
didn’t study yet at PCORI. I think PCORI might have a real
interest in what those are. – Sure, I’m interested as well. (audience laughing) – So I think there are several lessons that can be learned from what
the CF Foundation has done. Some related to drug development, some actually independent
of drug development. I think the CF Foundation had
a major impact on survival of patients with CF
independent of drug therapies. And through their
identification of best practices and how to manage patients with CF. But apropos drug development,
I think the patient registry is a useful tool. One, for learning about natural history. We keep talking about external controls. For two, identifying
subjects for enrollment in clinical trials and then three, for a collection of real-world
data and real-world evidence for primarily, like in this
case, for label extension. Whether it’s in rare
subsets or after accelerated approval, the confirmed benefit. So in a phase IV environment, such as a post-marking requirement. So I think that that
information is very useful and cystic fibrosis has been good. In a way, a platform trial or a platform is very analogous to a registry. And our folks in the medical device arena are way ahead of us. You’ve alluded to gastric
bypass or bariatric surgery. And they’ve been dealing
with this for a long time. There are a lot of lessons
that we could learn from our colleagues at CDRH, just for, maybe not fortuitous,
but for the circumstance that most devices are
approved outside the US and are used for many years before they get on the market in the US. And therefore, there’s
a wealth of experience in clinical practice. So they use registries very frequently for demonstration of efficacy of devices. So I think there’s a lot we
can learn from these lessons. And it’s definitely important. I think cystic fibrosis has certainly served as a prototype
of how we could do this in a variety of diseases. – All right, you all have
heard a lot of really insightful remarks from
this panel about next steps from what we’ve heard talked about today. Really some interesting ideas about extending this patient-focused platform into other areas of real-world
evidence development. Before we wrap up, this is a chance for final questions, comments,
for the panelists about any issues that
you’ve heard about today that you’d like to follow up on. Any ideas, thoughts about next steps. So I’d like to open this up
to questions and comments. And again, those of you who
are just joining us online, feel free to email us
about your question, too. – [Michael] Michael
Liebman, IPQ Analytics. Joe, you mentioned bringing
in some of the parameters from things like EHRs and you specifically mentioned things like ejection fraction. And bringing together
clinicians and so on. I think when you start to look
at some of these parameters, one of the things you need to look at is that clinicians will deposit parameters or measure parameters based on guidelines and procedures that they’re
trained in and used to applying. Especially if we’re trying to drive the integration or the merger of research and clinical practice, we need to look at the fact that things
like the echocardiogram or the ejection fraction as measured have a very large number
of additional parameters that never appear in EHR
and that, if you’re looking at a cardiac patient,
a cardiac physiologist has a completely different
perspective on what’s important about heart function than a cardiologist. So it’s just an opportunity
as you’re planning, as you said, to make sure
that other perspectives are able to be incorporated
in that early planning stage. – Thanks.
– I think there’s an enormous amount of work that
needs to be done with EHR to make it fit-for-purpose. And we’ve talked about it all day and there have been
meetings, all-day meetings, just focusing on that one issue. I think one thing to point
out, sort of a tangent about what you said, but
related to patient-focused drug development, is that the endpoints that we use right now. So HAM-D scores for depression, pain scores for treatment of pain. SF whatever, 26 or
whatever the number is for. But all these things
are not used in standard clinical practice and are not there. When you start working to change the EHR, if you can actually change
the EHR in the entire country, to a uniform data set, as you do that, you have to keep in mind that we may be changing our endpoints. To, someone said earlier,
whether they could be home or at graduation or whatever it was. But it’s different from what we’re using in standard clinical trials. We have to keep that in mind. It’s a lot easier to adjust a CRF for an individual trial to say, okay, this is going to
be our primary endpoint. How someone feels about
fatigue is what we’ve learned is the most important
symptom of this disease for these patients with this disease. We’ve been using some
other measure and then all of the sudden, that’s not in the EHR. And this is most of what
we’re talking about doing, at least for observational studies, is probably going to be retrospective. So it’s not going to be there. It’ll be a missing element. So it’s just something to consider as we roll up our sleeves and start doing all the work that’s needed to be done to get the EHR fit-for-purpose for real-world evidence generation. – Thank you, other questions? It looked like.
– Yes, hi. Jennifer Christian, I’m Vice
President of the Clinical Evidence at QuintilesIMS and
I have three quick comments. The first is my group works
a lot on innovative studies, pragmatic trials, those
that combine prospective data collection with existing databases. As well as extension
studies, so those that roll over patients from
RCTs and follow them for long-term benefits and risk. And one of the big questions we get, as we’re working with
pharmaceutical companies are what evidence do you
have that this design has been used before for
regulatory decision-making? A lot of case studies were used today, but I would encourage one of the outputs from this to be a way of sharing these more systematically and describing some of the context that
went into the reason both for using it and
the impact that it had. The second comment I
have is around endpoints. And I love Joe’s comment
that they need to be really what’s happening in practice. So I would just give an
example in working in Crohn’s disease, the traditional
endpoint that’s often used to measure remission
is the Crohn’s Disease Activity Index, or CDAI, which requires seven days of a patient
diary prior to coming in, a number of biomarkers to measure, and then the point at which the
clinical evaluation is made. And if you ask
gastroenterologists, they say, we don’t use this in practice. Could we use something
that’s much simpler? Maybe a five or six question tool like the Harvey-Bradshaw index? In which case, maybe we could use a pragmatic study design then. Or look at reduction in
surgeries or hospitalization, in which case we could use claims data. So a big a-ha I think
in the planning of these is what endpoints can we use that then drive decisions around the design? And then my final point is just to plug, really, pre-specification
has been brought up, and I saw InSet being mentioned, there is ROPER, the registry
of patient registries where this could be listed. And we have been with HRQ on a new ebook that’s coming out this fall called 21st Century Patient Registries. Which, Preston, it does
include working with patients throughout the study design
and conduct of the study as well as designing more patient-centric studies and using digital technologies, so look for that. – Great, thank you very
much for the comments. I had a question over here and then over there, too. Where’s the microphone at? Yeah, go ahead. – [Lisa] Lisa Goldstein
with the American College of Cardiology, I have two questions. One, I’ve heard a lot
today about how patients want to contribute their
data and want to participate. And I don’t necessarily
disagree with that, but one of the things that we require in this country is patient consent. We keep talking about putting together different data sets and existing data sets for different purposes, but we’ve yet to solve the problem of consent. So how do we propose to address this issue of consent for ongoing future use while ensuring that
patients remain engaged and aware of how their data is being used? So that’s the first question. And then the second question is, a lot of what we’ve talked about today seems like it would fit
purposes for academic medicine, but not necessarily those
in private practice. Yes, they have electronic
health records that they hate, where they feel like
they’re clicking different, constantly clicking all
these different data elements and don’t understand why
they need to do that. So how are we going to keep
it to a manageable amount of number of clicks so that they can still engage with their patients properly? And provide appropriate
care while ensuring that we’re not developing a framework that only works for academic medicine. Injecting an entirely different bias into that data set? – Great questions, and we have
to keep the answers short. We did talk about informed
consent approaches a bit earlier today,
but any brief thoughts? – Yeah, I’ll skip informed consent because I’m not a lawyer. In terms of the last thing,
you’re absolutely correct. We have to decrease the
burden to physicians in the EHR, so by modifying the EHR, we don’t want to make it worse. What we want to do,
though, it is make it fit, if you make it fit for clinical practice, it’s going to be fit
for research, in theory. So they will then like it. Right now, they don’t like
it, because right now, it’s a tool primarily
used for reimbursement. And it’s one that they do begrudgingly because that’s how they get paid. I think again a lesson learned, and I’m not sure that this may
be transferrable to the US, is the Southwark project. And they found that in having
pharmacists and physicians enlisted and that they knew
they were participating in the research, this
was real-world evidence based on one drug, but
for two big studies. One on COPD and the other on asthma. They found that they had greater buy in. So they were enthusiastic to participate and then the side benefit
potentially, is greater adherence. One of the problem we have here in the US, like with atrial
fibrillation as an example, there are many patients
who should be prescribed anticoagulants who aren’t. And then once they’re prescribed, they don’t necessarily keep taking it. So there’s hope, I don’t know if there’s a definite it will
happen, but as physicians participate in these studies and know that the evidence was
generated from their patients and their practice, maybe
there will be better uptake of good clinical practices. – Joe?
– In response, I’ll take the first part about the IRB. So we’re here talking
about real-world evidence and I really appreciate
the fact that the FDA has said several times
that real-world evidence often comes from randomization. But sometimes it doesn’t. And in the case of registries,
it typically doesn’t. So I would say that in terms of HIPAA and the constraints on the use of the data for research purposes that
could not otherwise be done, I think that the use as well, arguably perhaps, linkage
is covered by HIPAA. But we have not had the conversation with the public about this, so
I think there will be people who raise the issue that you just raised. And that’s part of this societal change. – That’s what you were just talking about, some next steps on that.
– We simply haven’t ever had that conversation,
coming from Kaiser, places that use the
data a lot didn’t really talk about it a lot until
the publication came out and then it was a good time to celebrate. – So great questions, we
are about out of time. So I think we have time for
maybe two more quick ones. There’s one there, one
here and then one there. – Yes, thanks.
– Sorry, over here, first, – [Solomon] Hi, so my
name is Solomon Iyasu, I’m with Merck, great discussion. I wanted to have some maybe commentary from the panel on what
I see as the fundamental paradox in our evidence generation in the lifecycle of a product. As you know, drugs are approved based on the narrow indication clinical endpoint that is validated and for a short time. There are a lot of limitations and they are approved
for their purported use and benefit and under
specified conditions of use. Which is very different from
what happen in real life. But as we grow into
the post-approval phase we’re learning more and more about the safety profile of a drug. Both labeled and unlabeled
use under conditions of use approved and unapproved. So there’s a formation
asymmetry in the evidence generation landscape
that builds up over time where we’re getting more
information about safety which we have a certain amount of comfort. Although they are coming
from real-world data, but we are not developing the same type of information to capture the full range of benefitw that probably wouldn’t– – Including effectiveness and so forth. – [Solomon] So what should it look like in the evidence generation landscape to really characterize and capture all the safety and all the benefit profile apart from what is
measured in clinical trials to really define the
benefits of a drug as we go forward that’s acceptable
in a regulatory sense. – That’s the dream of real-world evidence. Is at the time of approval of any drug, there remain literally dozens
of unanswered questions. We sometimes don’t even
know what the right dose is. Based on the development program. Potential expansion or
limitation of the indications. Bob’s upset that I’m showing
our dirty laundry in public. – It’s so true.
– So there’s so much more we could be learning about a drug. So we have an option,
one is FDA could say, well, we won’t accept
any information except if it’s a randomized controlled trial for these literally dozens of questions. And so therefore, they don’t get done. So these questions remain unanswered. Or we could say we’ll consider
other sources of evidence, observational, whatever,
to inform labeling, clinical practice, et cetera. – And new kinds of randomization in this real-world pragmatic context, too. – Or not.
– Or not. – [Sheila] Hi, Sheila Weiss from Evidera. And it’s been a great day and I thank you, this panel and the others. And again reiterating, FDA
uses real-world evidence every day and they’ve
used it for a long time, particularly heavily on safety. But what I’m hearing here
is kind of a broad sweep, where are we going to
place this type of data in regulatory areas where
it hasn’t been used? For example, drug approval, and we know that definitely
within orphan drugs, that is happening more and more. But where else can it go and how far can it go down the line? And I think what we haven’t discussed is where something, a
question, is regulatory, for something that is being submitted to a regulatory agency for a key decision. And when is something
that’s a public health issue, where we’re looking at comparing drugs or the usefulness
or the side effects of drugs and how they’re
valued by the patients among all the treatments
that are available. Those are two very different and both very important questions. I see Marc Berger shaking his head there. But I think we need to
think about it that way, because the issue of approval is what is the quality of the evidence? And sometimes we don’t know the quality of the evidence until we get
the answer from the study. Because obviously, the
larger the difference, the more we are willing,
in most cases, to allow the rigor to be less than
if it’s a small difference. So I’d like to hear the
panel’s take on that. – And again, it does come back to some of the challenging core issues that we’ve talked about today. So any final, quick
comments on this question? Or any final comments on the panels? This is your last chance. – You raised several questions. And the one about when
do we believe an effect, direction and magnitude,
based on observational data versus randomized controlled
trial is a thorny one for us. If you saw it, we use it
all the time for safety. We’re not required by
statute to have substantial evidence of a safety finding. And we struggle, even
with the Sentinel database and the 190 plus million people in it. We struggle with safety signals, behind closed doors, scratching our head, trying to figure out is this real? Is it not real? Should we put it in the labeling? Should we not put it in the labeling? It is very difficult when
you don’t have randomization. Where you can establish causal inference. Because that takes care of both the known and unknown confounders. And I just want to say
that that’s going to be a very difficult thing for you all and for us going forward. And that’s why we keep on stressing, you can do randomization
within the healthcare system. – And appreciate the
collaborative approach to try to address these
issues of observational data. Any final thoughts from
the rest of the panel? – Just following up on Jonathan’s, I think we should reconvene in two years and ask that question, have
we gotten real-world data and the methods and the approaches and the capacity to randomize
in place sufficiently that we can now begin talking about using real-world settings
for regulatory trials. – Including on effectiveness, yes. – On effectiveness, yes. – Other final thoughts? – Yeah, I would just
say on this last point, and it was brought up before, this question about reproducibility. Is it of the methods or of the results, and I think it’s both. You shouldn’t be doing
research unless your methods aren’t transparent
enough that somebody else can actually do the research. So you have to be transparent with the methods that are being used. But when it comes to the
observational studies, that we’ve been talking about today, consistency of results
in multiple studies. So as Marc, the example
of four different studies in completely different data environments, that you’re seeing consistent
results from study to study, that means something
and that had some value. So I think a good step forward is trying to get all of you, our
thought leaders, together to think about, okay, what
are the appropriate ways to make sure that our
observational studies are credible? Consistency of results,
reproducibility of the methods, pre-registering a priori the protocol and the analytic plans and sticking to those are examples, but we need more information on what
the best approach is. – Preston, last word? – Last word, big pressure. With regard to this
specific last question, as I said, I am not an
expert on methodology. Again, I just have to
reiterate that turning it over to patients and
having the patient-centered approach is the best policy here. Keeping in mind that whatever we do, it should protect the privacy of patients. It should protect their safety. And it should be really
geared into the mindset of what is going to make
their lives day-to-day a better place? – Yeah, answering the questions that they want answered. Thank you all very much. (audience clapping) (Mark drowned out by clapping) For closing, today has been one step in what, as you’ve all seen,
is a big ongoing journey to getting answers to
the kinds of questions that Preston was talking
about in terms of using and leveraging real-world evidence. There clearly is a new level of interest, new capacities and hopefully
coming out of all this, hopefully within less than two years, Joe, some concrete steps that we’ll able to see as progress and impact from the kind of work that’s been done today. But I’m going to turn this over to Greg for thanks and next steps. So that suffice to say,
we really appreciate all that you’ve all have
contributed to this effort. And in advance, for the next steps in making progress on real-world evidence. – Okay, I’ll do that right now, so just stay here and I’ll
just do it from my seat. I actually didn’t know that
I was doing closing remarks. So this is the first time that I’m reading these talking points. You’re here, why don’t you do it? (audience laughing) Anyway, I want to thank everybody today for participating, the 1,500 people online and I’m sure due to our tweeting, we probably got up to 1,600 of the folks dialing in for the webcast, thank you for doing that. I do want to mention that we are going to have an informal comment period. So starting from today
and then 30 days from now, so up until October 11th,
we will be accepting comments through email to all of you to that email address
that’s on the screen. We’ll do our best after that 30-day period to synthesize that information and put out something in conjunction, or an addendum to the white paper,
summarizing some of the major comments that we heard, so I encourage you all to submit your
feedback and comments to our white paper through that route. I’d like to specially thank the FDA, not just for helping us with
this particular meeting, but we’ve been working
with you very closely over the last, two years or so, when we started talking about this topic. And we’ve been very appreciative to all of the comments and feedback and participation in the activities resulting in the white
paper and today’s meeting. I’d particularly like to thank
Jacqueline Corrigan-Curay, Dianne Paraoan, Melissa
Robbs and Kayla Gavin for a tremendous amount of work and you (laughs) for all of the
work that you’ve done, Jonathan, Rich Moscicki, Jonathan Jarow, who’s sitting right next
to me, representing FDA. For the Duke-Margolis
staff, Morgan Romine, who’s done a tremendous
job working with all of the FDA folks on our white
paper and making sure that we’re covering all
of the bases as well as designing today’s event,
which was a huge success. Katherine Frank, Ellen de Graffenreid, Elizabeth Murphy and Sarah Supsiri. So thanks to all of your participation, please provide us your feedbacks. And have a great rest of your afternoon. (audience clapping)

Leave a Reply

Your email address will not be published. Required fields are marked *