In the face of fraud, usual review methods are powerless. Raw data and scepticism may help avoid another debacle.
As you’ll no doubt have noticed if you’ve been following me for the last few months, there’s a big problem in the scientific literature.
After a lengthy investigative project, myself and a group of colleagues have discovered a deep well of fraudulent research that underlies the entire literature behind ivermectin for covid-19. Not just one or two problematic papers, but a staggering volume of studies that appear to either be so fatally flawed that they cannot be trusted or research that literally never happened at all.
We have just published an article in Nature Medicine about it.
It’s been a pretty wild ride. I suggest checking out parts 1, 2, and 3 if you are interested in catching up on the story. I’ll wait.
Even worse, I’ve currently got parts 4, 5, and 6 sitting in my drafts folder waiting to be put online — you have to be really careful in investigations like these so it takes a while to get them out even when you’re pretty certain of the facts.
Now, the thing about ivermectin is that serious, genuine academics believed that it worked just a few months ago. As I discussed, real meta-analyses by very competent PhDs came out arguing that ivermectin had a huge benefit in the treatment of Covid-19 and should be adopted throughout the world. People put on their metaphorical white coats, followed the mandated steps of the scientific methods, and came up with a conclusion that on face value appeared to be entirely correct. They put together systematic reviews and meta-analyses where they collected all the evidence on ivermectin and Covid-19, and concluded that it was probably an effective treatment for the horrible infection.
Until the fraud was revealed, and we realized that it might all be complete nonsense.
All of this brings us to a fairly terrifying fact— the process by which we gather knowledge to derive medical treatments, which might literally decide whether you live or die, has massive flaws. To anyone who has looked into issues in the medical literature before this is not at all surprising, but the degree to which we have failed during Covid-19 to properly vet treatments and their benefits is a depressing story of its very own.
Trust No One
The problem is, science ultimately relies on trust. As researchers, we rely on other scientists to conduct the experiments that they say they do, in the people they describe, and to do it all in a manner befitting the historical stuffiness on which all scientific enterprise is based.
This goes double for meta-analyses which are, at their core, an aggregation of other people’s work. We all assume that published research is described truthfully, because without this assumption the entire system falls down.
And in the case of ivermectin, fall is exactly what it did.
As I’ve previously described, once you exclude the definite and probable fraud from the ivermectin literature, any benefits all but disappear. It seems that the entire conversation about ivermectin has been mostly driven by people simply fabricating scientific papers.
So what can we do to fix this? Unsurprisingly, the first step is to start being a little bit more critical about how we appraise research.
The Fix
Realistically, there is no simple, easy fix. There rarely is when it comes to complex problems.
However, there is one thing that we can do which might help prevent another ivermectin story from occurring in the future. Currently, we basically take it on face value that people have indeed done the research that they say they have, and leave it at that — traditionally, when we conduct a systematic review and meta-analysis, we rate the study quality, but we completely ignore any assessment of potential fraud.
In a system where we assume fraud is almost unknown, this is perhaps understandable. If you think that fraud is a massive outlier and, like many scientists, believe that no one would ever fabricate a clinical trial, then it makes sense to never assess whether this has happened. Why bother with extra work when we know no one fakes research?
Unfortunately, in the real world, we know that fraud does happen. In fact, when it comes to ivermectin for Covid-19, it seems that fraud is far more prevalent than anyone would’ve imagined.
Which brings us back to the fix — essentially, we assume that every study is fraud until proven otherwise. This may sound harsh, but it’s actually not that unpopular a viewpoint, with a recent editorial in the British Medical Journal making exactly this argument as well.
What my colleagues and I who have been involved in the ivermectin debacle argue is that we should be asking every single study to provide basic data for their research to independent third parties to audit these studies for fraud. Most fraud checks are pretty simple, can be done on totally anonymous data, and overall it’s a pretty quick way to establish whether a study is likely to have occurred or not.
Now, this won’t catch every faker. Some people will slip through the cracks, because no system is perfect. However, even this simple, basic requirement would’ve caught almost every incident of fraud that we’ve come across. It’s very hard to fake a trial convincingly, and frankly most of the frauds haven’t bothered making datasets that would pass even the simplest of tests.
We wrote up this perspective and it was recently published in Nature Medicine, but without the academic language the basic message is simple — we cannot keep assuming that fraud never happens in science. This does not mean that no medical advance is reliable — if nothing else, most covid-19 vaccine studies appear to hold up extremely well and are very unlikely to be fraudulent — but it does mean that we need to view the literature with a slightly more skeptical eye rather than taking it at face value. In the case of ivermectin, it appears that hundreds of thousands, perhaps millions, of people have been treated with a drug based on studies that may never have happened at all.
If we want to keep this from happening in the future, something has to change.
Gideon Meyerowitz-Katz is an epidemiologist who tweets @GidMK
This piece was originally published at Medium
