Episode 45
Messy Data, Real Answers
In a world teeming with health data—from smart watch accelerometry to millions of hospital system electronic records—how do researchers find out which medical treatments truly work? Biostatistician Rebecca Hubbard discusses the messiness of real-world data, the limits of randomized control trials and how both of these powerful—but imperfect—methods are essential for building trustworthy evidence in public health.
Transcript
In the past few years, the field of public health has become more visible than ever before, but it's always played a crucial role in our daily lives. Each month, we talk to someone who makes this work possible. Today, Rebecca Hubbard
We turn to research to make sense of the world around us. Research shapes our country's health care policies, guarantees that vaccines are safe and effective, and helps us make decisions to stay healthy.
But how does research give us these answers? And how can we trust that it's giving us the right answers? To find out, I sat down with an expert: Rebecca Hubbard, a statistician and professor of biostatistics and data science at Brown's School of Public Health.
[:[00:00:57] Rebecca Hubbard: Thanks for having me.
[:But at the same time, she's been figuring out how to answer those questions–the best ways to conduct research.
[:[00:01:47] Megan Hall: So you could just like continue to learn new things while refining the methodology.
[:[00:02:15] Megan Hall: When a biostatistician is researching a medicine, or health care method, there are a few ways to study it. They can do a very specific type of experiment, called a randomized control trial to see if this new treatment works. But another option is to use data from the real world. They can look at that data, study it for trends and figure out if the new treatment is effective. That's how Rebecca does her research.
[:And I was like, okay, let's take a step back and think about what we need to do in order to make these studies really high quality, valid studies that people can put their confidence in.
[:[00:03:22] Rebecca Hubbard: For instance, accelerometry data from smartwatches, social media data from people's social media posts, environmental monitors that describe air pollution and things like that. Data are everywhere now because of the fact that everything is electronic.
[:[00:03:48] Rebecca Hubbard: The important thing to keep in mind with health care derived data is they come from the health care system because people are coming in to receive health care in the way that they usually do. So that means the data that we have access to, are the data that you need in order to deliver care and also to bill for that care. They're not necessarily the data that we need to do research.
[:[00:04:27] Rebecca Hubbard: So in statistical terms, that creates all kinds of bias because you have differential information for different people, which is always problematic in research, and the quality and amount of information you have is possibly related to the outcomes that you're trying to study. So you're more likely to capture those outcomes for some people than others, which is very problematic from a methodological perspective.
[:[00:04:50] Rebecca Hubbard: The first and most important thing is to realize it's a problem. One thing that I observed, particularly during the pandemic when people were getting very excited about using real world data, health care data to answer these questions, is they were just like: We have so much data. We have data on 10 million people. If you dump it into an algorithm, you must get the right answer, and that from a statistical perspective is completely wrong. Having more data does not eliminate bias.
So the first thing I think is recognizing, okay, these data are not the same as research data. What I like to do is think hypothetically, if I had conducted a designed research study, what data would I have collected? What do I really need to know about everyone? Why are they receiving a treatment? How would I have collected their outcomes? And then take the health care data and try to harmonize it.
And once you do that, you start to see the gaps.
[:[00:05:52] Rebecca Hubbard: So in a randomized controlled trial, we wanna compare two different health care interventions, often two drugs or two surgical procedures, two treatments of some type, head to head. So we'll randomize patients to receive one treatment or the other.
There are a few key elements of randomized controlled trials. One is the randomization, so that's really, really important that the treatment or the intervention is going to be assigned randomly. That means that the people who receive each of the two treatments should, on average, look just like each other.There shouldn't be any systematic differences between those two groups.
In addition, it's important that there's a control, and the control will often be either a placebo or in real world data studies, what we're often interested in is the standard of care. So what's the standard treatment that people are currently receiving? Let's compare that head to head against some new proposed experimental intervention.
And that's really important to account for placebo effects, that people just generally do better when they think they're receiving something new and exciting and different.
[:[00:07:06] Rebecca Hubbard: “Traditionally randomized controlled trials are considered the gold standard.” I can't tell you how many papers I've started with that sentence. “Randomized controlled trials are the gold standard for medical evidence, blah, blah, blah.”
They're considered the gold standard for single study evidence because randomization eliminates confounding. So there's no systematic differences between people getting the two treatments, which is really important and really difficult to replicate without randomization.
Additionally, the quality of the data is high and pre-specified. The pre-specification is very important. So it's not just that the data are quite complete and there doesn't happen to be much missingness, but that prior to the conduct of the study, the researchers sat down and said, this is the exact information that we need in order to be able to conduct this study.
Here's the outcome, how it's defined, when it's going to be assessed, and we're gonna do that uniformly for everyone in the same way. I will say, despite all the papers that I've started by saying, RCT is the gold standard. You can do a really good RCT that produces high quality evidence and you can do a bad RCT. So each one has to be evaluated on its own merits. How well was the outcome defined, how complete was ascertainment, et cetera. so they are regarded as the gold standard, but they have flaws too.
[:[00:08:33] Rebecca Hubbard: From my perspective, the biggest limitation of RCTs is lack of generalizability. Because they are conducted in a really precise, specific way, that means the care that patients are receiving in the context of an RCT does not necessarily look like the care they would receive in routine practice. So often the results that are observed in a clinical trial population do not fully translate into a real world population.
In addition to differences in the care environment and the intervention that patients are receiving, there are also big differences in the patient population. I've seen this a lot, especially in oncology, where the patient population that's specified for an RCT, is the population that the investigators think has the highest likelihood of benefiting from the intervention, and that usually means they're healthier, and more likely to have a good prognosis.
Once the RCT completes and it shows this is beneficial and the drug is approved, and it goes out into the real world, everybody's gonna get it. So not just the good prognosis patients, poor prognosis patients, patients with barriers to accessing clinical trials who are underrepresented. And so there can be pretty big differences between the results that you see in the real world and the RCT results.
[:[00:09:58] Rebecca Hubbard: So to me the major advantage of using real world data is specifically that it addresses that generalizability limitation of RCTs.
So real world data are data that are generated as a byproduct of health care, and therefore they reflect the real patient population who will actually receive this intervention as it's delivered in the real world. That's a huge strength. There are huge limitations as well and they kind of go along directly with the strengths of RCT.
So the strengths of RCTs are the randomization and the high quality data. The limitations of real world data are no randomization. So now you just see the patients who happen to get the treatment that they happen to get. And the data are not pre-specified. So whatever data needed to be collected to complete this patient's clinical care and billing, that's what you have to deal with.
[:[00:11:04] Rebecca Hubbard:I do definitely think about that, like when I'm analyzing electronic health records data, I think like, I could be in this data set, or my partner could be in this data set. And that makes me want to work on the ethics of using these data responsibly to make sure that the students and trainees that I'm working with think about the ethics, and really treat it as every data point is a person and every data point should be treated with respect. Because it's a gift that we're being allowed to analyze these data.
[:[00:11:38] Rebecca Hubbard: Probably the most important one is the concept of minimum necessary– that we want to access only the data that we need for research, and nothing beyond that. We certainly don't wanna go poking around in people's medical records, you know, finding out information about them that we don't need. So when I'm conducting a study with electronic health records or claims right at the start of the study, I'll do that hypothetical exercise of thinking through, okay, what would a designed research study look like using these data? And then those are the data that I'll request. So I request only the information that I need, and I always want to see as little about people's private information as I possibly can.
So for instance, I as a statistician, do not need to know your name or your social security number or your address. I never want to see that kind of personal, private information. It makes it harder for me to work with the data if it has identifiable information, but also, it's a breach of trust to get access to things that we don't actually need.
[:[00:12:50] Rebecca Hubbard: Yes. So about five to 10 years ago, I was working with oncologists at the University of Pennsylvania, and at that time, immunotherapy had just been approved for treatment of advanced bladder cancer and immunotherapy had been approved through this pathway that the FDA has called accelerated approval.
In accelerated approval, you don't have to do an RCT to get your new drug approved. So immunotherapy had come on the market, but there was no RCT data comparing it head to head with the existing standard of care, which was chemotherapy. So the oncologists came to me and said, this is on the market now. We are using it, but we don't actually know if it's benefiting our patients. There is an RCT that's ongoing to answer this question, head-to-head comparison of immunotherapy versus chemotherapy, but it's not expected to complete for some time. And in the meantime, they had this evidentiary gap where they're like, what should I do? What's best for my patients? So because immunotherapy was being used in routine practice, we were able to take oncology EHR data, look at the patients that were getting immunotherapy, compare them to patients that were getting chemotherapy, do the head-to-head comparison in the absence of the RCT and generate real world evidence about how patients were doing.
What we found was immunotherapy looked worse than chemotherapy early in follow-up, but if you followed patients out long enough, the patients who survived over time ultimately did better on immunotherapy than chemotherapy. About six months after we published that result, the RCT completed, they published their results and their results are amazingly similar. So I like to show a slide in my presentations where I'll put up the RCT result and the real world data result, and they followed the exact same pattern.
So we were able to fill the evidentiary gap, advance the evidence earlier, answer this question for oncologists. Ultimately the RCT came up with the same answer, except you can see in the two plots that survival overall in the real world is quite a bit lower than in the trial. So the trial patients were healthier, better prognosis, the relative performance of immunotherapy versus chemotherapy is the same message, but you can also really see that in the real world, patients are different, patient outcomes are different. And so I think that was important to contribute as well.
[:[00:15:30] Rebecca Hubbard: So I would definitely not conclude that we don't need RCTs. I really think of all the evidence that we generate from these different study designs as going together in complimentary ways to build an edifice of evidence that we consider in its totality rather than any individual study.
So we can pick out the RCT and say, you know, I'm worried about the generalizability of this study, you know, it has certain specific flaws. We can address those flaws with real world data, but there are biases in real world data studies that are unsolvable.
The confounding due to lack of randomization: you can never be completely confident that you've solved that problem, that you collected every patient variable that was important to account for. So I think we really need both. We need to do the real world data studies sometimes to confirm that the RCT evidence actually translates to the real world.
We still need the RCTs. We need the high quality data. We need that kind of bedrock of evidence so we can be confident. And I really wouldn't trust any one study in isolation all by itself. I think you need all of this evidence taken together to come to an answer.
And I also think in science, science is self-correcting. So sometimes when I present about electronic health records and real world data, people's reaction will be, but the data are so messy. How do you know you're getting the right answer? And my answer is we are doing the best that we can, the highest quality study that we can, but also acknowledging the biases and limitations and we're gonna do another study.
This is not the end of the chain of evidence. There will be future studies. If it indicates something different, then we will update the evidence, and continue to learn and move closer towards that right answer.
[:[00:17:38] Rebecca Hubbard: That's what I think. I know sometimes RCTs are called “pivotal.” So the pivotal RCT is supposed to be kind of the final answer to the question of, is this treatment effective? But ultimately, science never ends. The patient population changes over time, disease characteristics, disease risk factors change over time, you always need to continue to update the evidence. So yeah, the RCT sort of starts the conversation and then the RWD will carry it forward.
[:[00:18:02] Rebecca Hubbard: Thank you so much for having me.
[:Humans in Public Health is a monthly podcast brought to you by Brown University School of Public Health. This episode was produced by Nat Hardy and recorded at the podcast studio at CIC Providence.
I'm Megan Hall. Talk to you next month!
