(CNN) A little more than three months after the World Health Organization officially declared the coronavirus outbreak a global pandemic, it has become evident that both the research cycle and the news cycle have accelerated to levels never seen before in our lifetime.
According to the Milken Institute, there are at least 254 treatments and 172 vaccines currently in development to fight Covid-19. I've reported on many of them. Some of them are just being developed, like PAC-MAN, a new anti-viral treatment that uses the gene therapy technology CRISPR. Others are drugs finding a new life, like remdesivir, which initially showed effectiveness in treating animals with SARS and MERS, and was even trialed unsuccessfully for Ebola.
The media's coverage of these developments has also been at "breakneck speed" -- because finding any way to stall the spread of this disease is so imperative. For example, several scientists recently called me both on and off the record to relay their optimism that a vaccine could be available by the beginning of next year. It would be a remarkably fast process, given that vaccine development is typically measured in years or decades, not months. So this past week, I took a step back to dig deeper into the studies and look into the source of this optimism. I was surprised at how thin the available data actually is in peer-reviewed medical journals.
Truth is, most of what we have seen so far has come in the form of press releases or pre-print reports that have not undergone the scientific scrutiny of independent review. In fact, despite all the enthusiasm around vaccines, there is only one published study of a vaccine trialed in humans -- from the Chinese company CanSino Biologics.
There is no question that in this environment, speed is of the essence. Scientists are scrambling to learn about the virus, the disease and how to keep people from dying. Health officials are working hard to put sound public health measures in place that don't overburden society or shut down the economy. And journalists are running ragged trying to cover it all.
But there are also growing concerns -- on the part of scientists and journalists -- that the studies being offered up and showcased are not ready for "prime time." In fact, many are not studies at all, but subjective conclusions based on data, and methods that remain hidden and thus difficult to validate. Never before has full and immediate transparency been so important, and never before has the scientific picture around Covid-19 been so opaque.
Press releases, pre-print papers and published papers all serve different purposes, and carry different weight for both scientists and journalists.
A press release "is there to make your institution, your client, your big name researcher, your product, your drug company and its products, look as good as can be, hoping that that press release will convince journalists to write about it," Gary Schwitzer explained to CNN. Schwitzer is a longtime health journalist and publisher, and the founder of HealthNewsReview.org. Because it's written by whoever is promoting the product, it's almost never negative, Schwitzer said.
Traditionally, pre-print papers have been articles that researchers and academics put out on pre-print servers to get feedback from their peers before they submit their study to a journal. During this pandemic, the profiles of at least two of them -- medRxiv (pronounced med-archive), for health sciences, and bioRxiv (pronounced bio-archive), for biology -- have been greatly elevated. "Pre-print servers are much, much more important than they ever have been in Covid-related areas -- in other words, in life sciences, in clinical medicine. They just weren't a player before this," Dr. Ivan Oransky told us. Oransky is the co-founder of RetractionWatch.org, Vice President of Editorial at Medscape, and a medical journalism professor at New York University.
A study published in a credible scientific journal is -- in theory -- the final, complete version. To get published here, a study has to undergo a process called peer review. Kate Grabowski, an assistant professor in the department of pathology at Johns Hopkins University, calls the peer-review process "multiple, independent sets of eyes" on a paper. While peer review is by no means fool-proof, it typically reflects the expertise of many people in a particular field who don't necessarily have a "dog in the race."
"I think it's just so valuable to picking up potential errors that are largely unintentional, and also just making the science better. Usually when we submit papers, they're like rough drafts, and then may get refined [several times] until they're much better," Grabowski said. She described the process, to us, as "iterative."
But the past few months have highlighted that the road to solid science can be full of potholes, speed bumps, blind spots and hairpin turns. If you are not careful, sometimes that road can lead you straight off a cliff.
Here are several recent examples of the story getting ahead of the science:
HYDROXYCHLOROQUINE: By now, you have surely heard of the anti-malarial drug hydroxychloroquine. But, you may not have known that at the time President Trump touted its benefits, there was a pre-print report from French researchers originally posted online on March 20. That study was of just 42 patients, and it has been criticized for confusing data, poor protocols and not clearly accounting for all the patients' outcomes, according to Oransky's Retraction Watch website. On April 3, the International Society of Antimicrobial Chemotherapy -- the society that publishes the journal the paper was supposed to publish in -- issued a notice saying, "the article does not meet the Society's expected standard."
On May 11, another study on hydroxychloroquine appeared on medRxiv. Nine days later, the study was withdrawn. In the abstract is now a statement saying: "The authors have withdrawn this manuscript and do not wish it to be cited. Because of controversy about hydroxychloroquine and the retrospective nature of their study, they intend to revise the manuscript after peer review."
Unfortunately, none of this stopped the hype around the drug, leading to critical shortages for patients using it for the effective treatment of lupus and rheumatoid arthritis. As more thorough published research on the drug finally started to emerge, it was shown to work neither prophylactically nor as a treatment for Covid-19, and its side effects were found to be potentially dangerous to certain patients. The US government now has a stockpile of 63 million doses.
DEXAMETHASONE: More recently, study results on the commonly used steroid medication dexamethasone were released via press conference first, frustrating researchers who wanted to see the hard data upon which these good-news findings were based. Harvard health expert Dr. Ashish Jha tweeted: "First -- it is now a feature of this pandemic that most findings made public via press release with little data to provide context. Second -- this is REALLY good news if it turns out to be true." The problem is that at the time we hear about these results in the public, we should have the confidence to say it is true. Unlike with hydroxychloroquine, the study results of dexamethasone appear to hold up. In patients on ventilators, mortality was reduced by one third, according to the pre-printed, early study, which was released the following week. But, once again, we still have to see if it will hold up to the rigors of peer review.
MODERNA: Moderna, one of the companies in the very competitive race to produce a vaccine against the coronavirus, set off a frenzy on Wall Street after sending out a press release on what some considered hyped-up partial results from Phase 1 of its trial. At the time, Dr. Peter Hotez, a leading expert on infectious disease and vaccine development at Baylor College of Medicine, told CNN he found the results of Moderna's press release "uninterpretable." "It didn't contain any data," he said. "So basically -- it's opinion. It was spin and opinion." When asked by CNN to respond, Moderna issued a statement saying: "The Company worked cooperatively with NIAID to properly characterize the interim data provided by NIAID. Moderna also said on its subsequent investor call that we expect NIAID and its scientific partners to publish full data from the trial at a future date."
The Moderna vaccine has been developed in partnership with the National Institutes of Health's National Institute of Allergy and Infectious Diseases, headed by Dr. Anthony Fauci. Fauci has often cited the speed at which the vaccine was brought to Phase 1 trials. "That is overwhelmingly the quickest that has ever been done," he said. He further added, "Given the fact that we needed to do this as quickly as possible without sacrificing safety or scientific integrity, the federal government partnered with multiple of these companies and said, 'Guess what? We're going to move fast and we're going to assume we're going to be successful. And if we are, we've saved several months. And if we're not, the only thing we've lost is money.' But better lose money than lose lives by delaying the vaccine."
Fauci is optimistic about the potential vaccine but was also disappointed by the early release of results. "I didn't like that," he told STAT News. "What we would have preferred to do, quite frankly, is to wait until we had the data from the entire Phase 1 -- which I hear is quite similar to the data that they showed -- and publish it in a reputable journal and show all the data. But the company, when they looked at the data, as all companies do, they said, 'Wow, this is exciting. Let's put out a press release.'"
In fact, when it comes to the vaccine race, despite all the talk you may have heard from Moderna or Oxford University or "Operation Warp Speed," or having billions of doses by the end of the year, remember: We've only seen that one study published in The Lancet. That's it. We need to remember to temper our hopes and enthusiasm with the facts we have.
I include these examples because they show how different parts of the process can break down.
"Trying to do science by press release, without backing it up, either with a traditional journal or a preprint ... has universally led to misunderstanding and has no place in science. The biotechs are doing it because they're writing for their shareholders, they're writing for their investors, but it's being done in a way that's oblivious to its public health impact and needs to stop," Hotez said.
Put another way by frustrated research scientist James Heathers in a tweet: "Science by press release is just promises with numbers sprinkled on it. GIVE. US. THE. GODDAMN. PAPER."
When this happens -- and especially when errors are revealed or papers are retracted -- public trust is eroded and people begin to doubt science.
"It's like whiplash," Oransky said. "I'd be really confused."
But nowadays that whiplash has become almost inevitable because that back and forth is how science moves forward in this Covid way of life.
Adding to the whiplash and confusion is just the sheer number of studies coming out. According to Grabowski, who based her estimate on the NIH's iSearch COVID-19 Portfolio, approximately 35,000 articles have been amassed to date on the topic -- and they keep coming.
To get a handle on all of them, Grabowski heads up a team of about 50 Johns Hopkins University researchers at the Novel Coronavirus Research Compendium, which curates and reviews emerging research. She estimates they screen between 1,500 and 2,000 articles per week, and have looked at more than 10,000 of them total. As you might expect, some are garbage and some are gems.
"I would say there's definitely some excellent studies that are being conducted under a really rapid timeframe," Grabowski said. "It's really amazing to see science moving at this pace. I don't think we've ever seen anything like this before."
All the experts said it is great news that so much research is being done, despite the fact that so much of the work is emerging through press releases and pre-prints.
"The fact that scientists are getting work out there earlier so that other scientists can pore over it and we can maybe learn things more quickly -- that's a good thing," Oransky said. "The fact that we're treating it all equally as if it's all been ... subject to the same level of scrutiny -- that's the problem."
Schwitzer cautions journalists and those who disseminate information to take the time to do it right. "Just reminding people to slow down. So much of what we're doing, reporting breathlessly at breakneck speed, doesn't need to be reported restlessly at breakneck speed," he said, adding, "Words matter and the data matter."
So, what does this all mean for you? "I think that someone reading, viewing, watching, listening should never make any decisions based on a single report they read, whether it's a study or a news report on a study," Oransky said. "Particularly if that news report doesn't put into context everything that's come before and doesn't explain what we still don't know."