With more journals adopting a ‘publish now, peer review later’ policy, even the most average science is gaining international attention
Sifting through my latest emails, I come across yet another scientific paper claiming to have found a successful drug for treating COVID-19.
You know what they say about something sounding too good to be true?
Reading down the page, some red flags quickly emerge: a small sample size; subheadings which don’t match the body text; and perhaps unsurprisingly, an admission the paper has not been peer reviewed.
And just like that, another study hits the recycling bin.
This is the controversial world of instant scientific publishing, which has seen an unprecedented uptick in recent months as researchers around the world scramble for answers that will curb the impact of the SARS-CoV-2 virus.
By mid-March, more than 24,000 scientific papers which mentioned COVID-19 had by-passed peer review and were publicly available for download from a range of online servers.
In less extraordinary times, most researchers would have happily waited months, or even years, for their work to undergo peer review and be published in a medical journal.
But it’s this same lengthy publication process which is making the instant publishing of draft manuscripts, known as preprints, all that more attractive.
Preprints are hardly a new phenomenon, the first online preprint server, called arXiv (pronounced “archive”), was created almost three decades ago for use by mathematicians and physicists.
But medical publishing managed to long disincentivise preprints by refusing to publish any manuscripts which had been preprinted without peer review.
This was until 2013, when dozens of preprint servers for the biological sciences, such as bioRxiv (pronounced “BioArchive”), started publishing thousands of draft manuscripts.
And the preprint publishing movement was brought further into the mainstream in June of 2019 when the British Medical Journal launched its own preprint server, known as medRxiv (pronounced “MedArchives”).
Fast forward to February this year, and with the global coronavirus outbreak, even medical journals which once followed a conservative publishing approach were forced to consider preprints as a way to share scientific information quickly.
In March this year, the Medical Journal of Australia (MJA) was the first Australian journal to launch its own preprint server, as a response to the COVID-19 pandemic.
Editor-in-chief of the MJA, Professor Nick Talley, told The Medical Republic that it was clear rapid access to science during this pandemic could save lives.
“It became clear in my thinking that we couldn’t run a standard process for getting what could possibly be really important data out for scientists and clinicians to utilise,” he said.
The MJA had traditionally operated a print and online model of publishing, with all papers undergoing full external review before being published.
“The idea of running a preprint approach and setting up a preprint server is something we did very rapidly,” Professor Talley said.
But unlike some preprint servers, which offered little in the way of quality control, the MJA was determined to keep some of its review measures in place.
GATEKEEPER MODEL
Professor Talley knew preprints were an advantage for quickly disseminating data to researchers and the private sector, but remained concerned that some preprints would be declared unsuitable following peer review.
“When that happens, you’ll never be able to publish the study in its entirety, which means you have a preprint but no article with a DOI (digital object identifier) which is the key thing an article has that allows it to be on PubMed and referenced in others work even more broadly,” Professor Talley said.
But the risks seemed minor to the MJA when compared with the possibility of receiving important papers about COVID-19 and not getting this material out in time for it to be helpful.
The compromise was a rapid review process, sometimes completed in as little as 24 hours from when a manuscript was received.
“Internally at the MJA we have a team of medical editors, we have biostatisticians and experts in scientific methods internally, so we do a rigorous internal review of the preprint and turn it around very fast,” Professor Talley said.
And if the internal review decided a paper still needed full peer review, more time would be taken to ensure an independent researcher could review the manuscript.
Other papers which are deemed unsuitable as a preprint can also still be accepted by the journal to undergo a traditional review process.
And the MJA has already seen the benefits of preprints, with one article about safe airway management, published in mid-March, having been downloaded more than 70,000 times.
“That’s what I call impact and relevance,” Professor Talley said.
“While it’s subsequently undergone peer review, at the time we just put it straight out with confidence that it was very good and expert commentary.”
A TEMPORARY SOLUTION?
But while reaping the benefits of preprints during the COVID-19 crisis, the MJA remains on the fence about making them a permanent fixture.
“I like the preprint model in terms of being able to get important papers out rapidly and I have been converted, but I still have some concerns I originally had,” Professor Talley said.
“We know there are preprints on COVID-19 which have received a lot of attention and have been found to have had a huge number of flaws. It makes you wonder if these preprints really should be out there. We certainly worry about this every time we do it.”
One study which attracted criticism in recent weeks was a preprint estimating how many people in Santa Clara county in the US had contracted COVID-19.
The study, examining the prevalence of COVID-19 antibodies in 3,300 volunteers, estimated that the true case count of COVID-19 in the county at the time of the study was 85 times higher than the recorded case count.
The study was picked up by hundreds of US and international news outlets before other researchers had a chance to read the study in full.
Upon doing so, Assistant Professor Will Fithian, a statistician at the University of California Berkeley, said he quickly noticed major mathematical errors in how the confidence intervals had been calculated, and immediately tried to get in touch with the study authors.
“I have been corresponding with the authors of the well-known Santa Clara County COVID-19 preprint, and I am alarmed at their sloppy behaviour. The confidence interval calculation in their preprint made demonstrable math errors – not just questionable methodological choices,” he said.
Professor Fithian said that while every researcher made mistakes, it was important for the scientific record to be corrected as soon as possible.
Everyone makes mistakes, but the record must be corrected ASAP. I emailed them on Saturday morning asking them to do so. In the last three days they haven’t corrected anything yet, but a subset of them have released a new study without saying how they did the analysis this time.
— Will Fithian (@wfithian) April 21, 2020
Professor Fithian said given the difficulty in perfect study design, it was all the more reason for authors to provide as much transparency as possible to their peers.
“Having done some applied work myself, I know how hard it is,” he said.
“Some mistakes are inevitable even given the best efforts to be careful. That makes transparency of methods, code, and (where possible) data all the more important.”
PEER REVIEW NOT PERFECT EITHER
But in the defence of preprints, Mr Gideon Meyerowitz-Katz, an epidemiologist at the University of Wollongong, said it would be naïve to think mistakes such as the ones in the Santa Clara paper were unique to preprints.
“I think at the moment we have some really interesting examples of very poor papers published very quickly, without peer review and while we may be seeing more of them, it’s important to remember that you get poor papers published all the time,” he told The Medical Republic.
“People often think of peer review is this wonderful process that tends to get rid of bad science, but really peer review just means the editors, and whoever they decide to send the paper to, gets to have a say in whether the article is scientifically valid.
“And given not all editors, and not all reviewers, are great, sometimes bad scientific papers get through the peer review stage.”
And the idea that bigger medical journals are more reliable is also ill-founded.
Previous research has found larger journals with a higher impact factor are responsible for around half of all annual retractions.
Researchers such as Mr Meyerowitz-Katz see great potential in being able to rapidly share evidence with their colleagues to critique, either formally or informally.
“In a usual situation preprints are great because they allow people to get rapid feedback on an article, with the entire idea being that preprints are not a final copy,” he said.
Mr Meyerowitz-Katz said there was a need for the medical publishing community to appreciate preprints for what they are – a draft which may have errors and are not an absolute truth.
The opposite has been happening in the midst of COVID-19, with small excerpts of manuscripts being shared around like they are facts.
KEEPING PERSPECTIVE
This is in part due to how the media has covered preprints during this pandemic.
One of the most controversial preprints has been a study that claimed hydroxychloroquine was a potential therapy for COVID-19 and made even more effective when used in conjunction with azithromycin.
That French study soon became the basis for hundreds of news stories. The supposed benefits of the antimalarial drug also became a talking point for US President Donald Trump over the first weeks of April.
The anti-hype soon arrived, in the form of a slightly larger study which not only proved the ineffective use of hydroxychloroquine in patients with COVID-19, but also showed it could lead to worse outcomes.
This cycle of misinformation reinforces what happens when journalists misinterpret the findings of preprints or misunderstand the nature of preliminary results.
Assistant Professor Ray Moynihan, an academic at the Bond University Institute for Evidence-Based Healthcare, said the advent of preprints has changed the way science was being reported in the public domain.
As a former health journalist and now someone who publishes research, Professor Moynihan has been on both sides of scientific communication.
“There are such dangers in widespread reporting of non-peer reviewed papers,” he told The Medical Republic.
Professor Moynihan participates in a weekly journal club with other academics at Bond University with the aim of critically appraising preprint manuscripts which are garnering media attention.
“One preprint that had come out in recent weeks was a randomised trial of one of the new potential treatments, and the drug was presented as potentially promising,” he said.
“But by the time we had finished a critical appraisal, we were all extremely concerned that the paper had grossly overstated the potential benefits, we were greatly concerned about the methods used.
“I don’t know how much how much media coverage that study has got, but I’d be extremely concerned if the sort of positive messages in that article were being transmitted through media, because that would be utterly misleading.”
While journalists are quick to report some of the exciting science being published before peer review, there are real dangers about interpreting information prematurely for the general public.
“If we’re going to report these things, and of course we’re going to report them because they’re publicly available, there must be big, huge bolded caveats right at the top of the story every time they report a non-peer reviewed study,” Professor Moynihan said.
Sometimes preprints will be reported in the news with only mild caveats, with journalists not wanting to undercut the value of their story.
“But I think if the media are going to report more of these preprints, we have to have much, much, more routine declarations of the caveat that this is non-peer reviewed,” Professor Moynihan said.
And while it may seem at face value that some studies are more reliable than others, not a single study published to date has been able to prove a drug’s effectiveness for COVID-19 in a large-scale double-blinded randomised controlled trial.
However, many such studies are in the works with more than 1100 clinical studies having been registered globally on clinical trials registries, including more than 500 RCTs.
Authors in a recent not-yet-peer reviewed study in the MJA said this rapid development in the scientific community would come with its own unique challenges, such as the potential for duplication and increased competition among researchers.
And until we start to see these RCTs published, we are still left to interpret the unfolding medical emergency through the patchwork of preprints from anonymous servers, and well-known journals alike.
So, for now, how do we know what information to trust?
This question is what led to the establishment of the National COVID-19 Clinical Evidence Taskforce.
This taskforce was set up as an Australian response to deal with fast medical publishing, with the goal to review available evidence and consistently update evidence-based clinical guidelines for medical practitioners based on their reviews of the research.
WHEAT FROM THE CHAFF
But it’s no small task to try and make clinical summaries from the thousands of papers which have been published on COVID-19.
Associate Professor Julian Elliott, the lead for evidence systems at Cochrane and the executive director of the new taskforce, told The Medical Republic the taskforce was having to screen 500 papers a day.
“Many GPs would be familiar with the typical guidelines that might be updated once every three, five, or seven years, or so. In the living evidence model that we’ve developed, we are now doing it now in warp speed and we’re updating weekly. That’s unprecedented,” he said.
“I don’t think anyone, anywhere in the world has really maintained guidelines at that level of currency. So there’s a whole set of challenges that go with that.”
Professor Elliott also sees the relatively recent advent of preprint culture in medical research, compared with other scientific fields, as responsible for some of the misunderstandings during this pandemic.
“This is the first time preprints in biomedical research are getting quite a bit of traction and you just have to recognise that this is a research output that hasn’t yet gone through peer review,” he said.
“You just bring that consideration into the way that you might interpret, and then use that information.”
Whether preprints will secure a comfortable place in medical publishing after the COVID-19 pandemic has passed remains to be seen.
But with no end in sight to this infodemic, the need to match fast publishing with fast critique through traditional peer review, taskforces, or less formal avenues such as Twitter, is proving to be more important than ever.
