Maybe someday there will be a vaccine or a miracle cure for COVID-19. But if that occurs, it will not be immediately obvious from a news story that’s trumpeted across your TV or your social media feed. Big, important-looking studies come along all the time, and most turn out to be disappointing once you learn the whole story.
Here’s a few questions to ask if you think you’re looking at something big:
What were the actual findings?
The headline is going to make things sound really dramatic, because that’s what headlines do. But go ahead and read the article, and ask the question: what did they actually find?
For example, one of the first headlines I saw on Twitter this morning was “Life-saving coronavirus drug has been found.” (This is the article; the headline has since been changed.) That headline implies that there is one miracle drug out there, we’ve now found it, and coronavirus deaths are a thing of the past. None of that is actually true.
Instead, the story explains that the drug reduced the mortality rate from 41% to 28% for patients on ventilators, and from 25% to 20% for patients who weren’t on ventilators but were sick enough to need oxygen.
The headline is accurate in the sense that lives seem to have been saved, but it’s not a complete cure, it doesn’t work for everyone, and it’s only shown these results in people who were already severely ill. Is that what you thought when you first saw the headline?
What do other scientists think of it?
Science is moving fast these days, and not every study stands up to scrutiny. Some COVID-19 studies have turned out to be shoddily done, or to overstate their conclusions based on minimal results, or to be based on data that seems sketchy and couldn’t be verified.
We don’t tend to find out about the problems with a study right away. It usually takes at least a few days of scientists discussing and examining a paper before problems come to light. (If you’ve heard of peer review, the process by which a few scientists are able to look at a paper before it is published in an academic journal, that’s meant to get a head start on this discussion. But it’s not a guarantee of quality.)
I’ve come to treat any new study or announcement with caution. As a reader, one thing you can look for is whether the reporter of a news article has spoken with scientists who have read the study, but who were not themselves involved in producing it. If every quote in a news story is from the people who conducted the study, be skeptical until you hear more.
Has the full study been released?
Anybody can call up a reporter and say they found some cool-looking results. If there’s science to back it up, though, they should be able to show everyone the full study. In the age of COVID-19, we’ve seen a lot of results announced by a press release or press conference, without the scientists actually publishing the full methods or the underlying data.
Ethically, the people who run studies have an obligation to act quickly if they find out that a treatment is either particularly dangerous, or particularly life-saving. The thinking is that ending a trial early, or releasing important results, can save lives and shouldn’t be delayed.
But there’s another side to that dilemma: what if the study is wrong, or the results are incomplete? Remember, you want other scientists to be able to look at the study to help determine whether it’s truly useful and what the limitations might be. Announcing a result without releasing data makes it nearly impossible for the rest of the scientific community to do its job.
Sometimes even non-medical studies are announced without a paper, like the story that half of Twitter accounts that discussed reopening were actually “bots.” The results were announced by press release, leaving other researchers to wonder how a “bot” was defined, and thus whether the study really found what the researchers say it did.
What comes next?
If we do find a true miracle drug for the coronavirus, it won’t be a one-off news story that is never seen again. The results will be confirmed with more trials, other scientists and doctors will comment on how it’s been working, and the story will reappear in the news time after time as that process continues.
Remember, there are dozens of drugs and vaccines in the works, and it will be a long time before we find out for sure whether any of them pan out. It’s probably more likely that we’ll get a few drugs that each help a little than that we’ll find a single miracle cure. So don’t just look at one story; consider where this new bit of news fits into what was already known. And wait and see what comes out next.
