Ivermectin and Statistical Significance

At what point should a potential therapy be deemed effective?                                          

Spring 2022            By Charles L. Hooper and David R. Henderson   for the CATO Institute

Since the beginning of the COVID-19 pandemic, humanity has been searching for effective treatments and preventatives for the virus. Responses have ranged from the wearing of facemasks and “social distancing” to the development of new vaccines and treatments. Researchers have also looked at existing medications to see if they can be helpful in combating the illness. 

One existing medication has received considerable attention recently: ivermectin, an antiparasitic that is widely used in the developing world. Many commentators, including several health officials, have dismissed the drug’s usefulness against COVID. Yet, these dismissals seldom cite empirical evidence, or if they do, they don’t detail the findings.

 Ivermectin, which is the generic name for the drug, was discovered in 1975 by William Campbell of the Merck Institute for Therapeutic Research and Satoshi Ōmura of Kitasato University, in work that would win them the 2015 Nobel Prize in Physiology or Medicine. Merck first marketed the drug as a veterinary antiparasitic (today it is best known by the brand name Heartgard), with human applications (and the requisite government approvals, under the brand‐names Stromectol and Mectizan) coming a few years later. In the developing world, the drug has proven so effective at combating parasitic illness that it is on the World Health Organization’s list of essential medicines. It has been dosed four billion times to patients in Africa and Central and South America. 

Ivermectin works through a variety of mechanisms to kill the targeted parasites. Some of those mechanisms have also been found to attack single‐strand RNA viruses like SARS‐CoV‑2, which causes COVID. That led scientists to test the medication in vitro, finding that it does in fact kill the virus in cell cultures. 

Because ivermectin has been around for decades, can be taken as an oral pill, is safe, and is now off‐patent and therefore cheap, it would be an ideal drug to give to COVID patients — if it is, in fact, effective in the body and not just in the petri dish. Is it?

Most Medical Authorites Say No

Most medical authorities currently discourage the use of ivermectin to prevent or treat SARS‐CoV‑2 infection. Because the drug is often used as a veterinary medicine in the United States and other developed countries, news that people were taking it to ward off COVID inspired the U.S. Food and Drug Administration last August to tweet: “You are not a horse. You are not a cow. Seriously, y’all. Stop it.” More thoughtfully, the FDA posted on its website, “Currently available data do not show ivermectin is effective against COVID-19.” .

Likewise, on its website, the National Institutes of Health posted, “There is insufficient evidence for the COVID-19 Treatment Guidelines Panel to recommend either for or against the use of ivermectin for the treatment of COVID-19.” The World Health Organization posted: “The current evidence on the use of ivermectin to treat COVID-19 patients is inconclusive. Until more data is available, WHO recommends that the drug only be used within clinical trials.” The American Medical Association posted, “The American Medical Association (AMA), American Pharmacists Association (APhA), and American Society of Health‐System Pharmacists (ASHP) strongly oppose the ordering, prescribing, or dispensing of ivermectin to prevent or treat COVID-19 outside of a clinical trial” (bold in original). And the U.S. Centers for Disease Control posted, “Be aware that currently, ivermectin has not been proven as a way to prevent or treat COVID-19.” 

Even Merck cautioned the public against the drug’s use against COVID. In February 2021, the drugmaker issued a statement that included this: “It is important to note that, to‐date, our analysis has identified: No meaningful evidence for clinical activity or clinical efficacy in patients with COVID-19 disease.” 

Yet, notice that these statements do not say ivermectin does not work against COVID. At most, they say only that, to date, research has not shown that it does work. So, what does the research show?

Early Clinical Data

At the time this article was written, the website c19ivermectin.com listed 73 clinical trials of ivermectin and COVID-19, involving 56,774 patients, as having been conducted. Thirty‐one of the studies (6,828 patients) were randomized, controlled trials. Fifty‐two were peer‐reviewed (18,768 patients). 

A few of the studies have been challenged and even retracted for shoddy work (perhaps putting it kindly), but most have not; we will look more carefully at these studies below. Still, the aggregate results are noteworthy. The treatment group had 59% lower mortality than the placebo or standard therapy control group (examined in 34 studies involving 44,061 patients), 48% lower use of mechanical ventilation (12 studies; 2,316 patients), 57% fewer intensive‐care‐unit admissions (seven studies; 21,857 patients), 45% fewer hospitalizations (19 studies; 11,190 patients), 71% fewer cases (13 studies; 11,523 subjects), 52% faster recovery (23 studies; 3,664 patients), and 57% improved viral clearance (22 studies; 2,614 patients). 

The FDA has approved many drugs based on less clinical research. When one of us (Hooper) worked at Merck three decades ago, the ACE inhibitor Vasotec (enalapril), one of the company’s biggest drugs, was tested in 2,987 patients before receiving FDA approval. The statin drug Mevacor (lovastatin), another of Merck’s big drugs at the time, was tested in 6,582 patients. Back then, that was considered to be a massive trial. 

Ivermectin was approved for the scabies and strongyloidiasis indications based on clinical data from 852 and 591 patients, respectively. Remdesivir was given an emergency use authorization by the FDA based on clinical data from 1,063 patients.

Reconciling Authorities’ Statements with Data

Given those results, it would seem the above‐quoted medical authorities are being overly cautious in their statements about ivermectin. Why would they do that? We can think of a few reasons:

• As noted, a few of the ivermectin studies have been of dubious quality. To some commentators, that tarnishes all the studies.
• Some of the studies were conducted in less‐developed countries where parasitic infections are common. Given ivermectin’s antiparasitic activity, the drug’s beneficial effects on COVID may come from freeing people’s bodies of parasites. The drug’s indirect beneficial effects on COVID would not be particularly helpful in developed countries, where parasites are less common.
• The FDA doesn’t have processes in place to properly evaluate drugs like ivermectin.
• Drug companies are rewarded for finding new therapies, not discovering new uses for old (i.e., off‐patent) ones.
• Commonly used levels of statistical significance handicap drugs that are not undergoing large clinical trials.
• Some advocates of ivermectin have little medical expertise — and a few seem outright kooky. This dissuades serious intellectuals from recommending the drug because they want to distance themselves from those who are less rigorous and objective.

Below, we consider each of these reasons in detail.

Studies of Dubious Quality

In July 2021, the preprint server Research Square withdrew a paper on a large clinical study indicating that ivermectin is effective against COVID. The withdrawal followed charges of plagiarism and faked data against the authors, Benha University (Egypt) medical professor Ahmed Elgazzar et al. Patient records appeared to have been duplicated and changed slightly and some of the patients seemed to have been hospitalized before the study even started. 

Elgazzar disputes these charges. He claims the patient data records at the center of the controversy were not supplied by him but were extracted by a researcher who hacked into his website, and that they weren’t the correct data records. Further, he said, Research Square gave him just 24 hours to respond to the charges, and that wasn’t enough time to investigate and prepare an adequate response. The dispute over the paper is ongoing, but it has tainted other studies of ivermectin’s effects on COVID. 

Some observers have claimed that once the results from Elgazzar and other questioned studies are removed from the body of research on ivermectin, meta‐analyses of the data from the remaining available studies find no benefit from administering the drug to either prevent or treat COVID. Those claims are incorrect. If we remove all questioned studies from the 73 noted above, that leaves 50 studies. Those 50 studies indicate that ivermectin still shows a benefit, averaged over several metrics, of 71% (range 61% to 77%). That is better than the 20% benefit (range 11% to 27%) from Gilead Sciences’ FDA‐approved Veklury (remdesivir). That is also better than the 34% benefit (range –13% to 62%) from Merck’s COVID treatment molnupiravir, which has received Emergency Use Authorization from the FDA. And the efficacy of molnupiravir in the latest Merck clinical trial, while less than originally hoped for (and less than a competing therapy by Pfizer), is still enough to materially help patients. As Peter Chin‐Hong, an infectious disease specialist and professor of medicine at the University of California, San Francisco, told the Wall Street Journal, “Thirty percent is still meaningful in a disease to prevent someone from being hospitalized.”

Only an Indirect Benefit?

Last November, Scott Alexander, a psychiatrist and author of the science‐heavy blog Astral Codex Ten (and, before that, Slate Star Codex), authored an extensive literature review of 11 ivermectin–COVID studies that he deemed to be of high quality. He tentatively concluded that, when ivermectin is given early in an infection, the studies indicate the drug reduces mortality by 40 percent, which is just barely statistically significant (significance: p = 0.04). Yet, he refrains from endorsing the use of the drug. Why? 

To explain why, he presents a hypothesis and a prejudice (more on the prejudice below). The hypothesis is what we noted earlier: ivermectin’s benefit may come indirectly by ridding the body of parasites. The relationship isn’t direct. It has to do with corticosteroids, which are a common treatment for COVID. When patients don’t have parasites, giving them corticosteroids generally helps. But when patients do have parasites, giving the corticosteroids can cause a medical condition called hyperinfection syndrome. Hence, by removing the Strongyloides stercoralis worm infections, ivermectin may prevent potential problems with corticosteroid therapy, leading to the conclusion that ivermectin helps with COVID..

 However, when the larger pool of studies is examined, they show a benefit to ivermectin of 72% in areas of low parasitic prevalence, while in areas with high prevalence the benefit is 55%. This is the exact opposite of what Alexander conjectured. Further, there is some evidence that the difference in the two areas can be partly explained by considering treatment delays — it’s better to give ivermectin early in the infection — and dosage size. In the geographic areas where the drug did better, it tended to be given earlier and at higher doses.

FDA Processes

The FDA does not grant approval to any drug until it judges the drug to be both safe and efficacious. On what evidence does it make that judgment? The drug’s sponsor must provide the agency a comprehensive body of clinical research on the drug’s effects. Such research is costly and time‐consuming to conduct. (That’s one of the reasons BioNTech partnered with pharmaceutical giant Pfizer to pursue FDA approval of its COVID vaccine.) 

For a drug like ivermectin, which long ago went off‐patent, no sponsor may step forward to finance and submit such research. The reason is not that the drug is ineffective, but that the sponsor won’t be able to capture much of the benefit from that research spending because competing generic versions of ivermectin are available. Because of generic drug substitution rules at pharmacies, Merck could spend millions of dollars to get a COVID-19 indication for ivermectin and then effectively get zero financial return. What company would make such an investment? 

With no sponsor, there is no new FDA‐approved indication and, therefore, no official recognition of ivermectin’s value. So, is the agency’s warning against using the drug as a COVID treatment based on science? No, it is based on process: specifically, the lack of a sponsor supplying the requisite data. 

Like other bureaucracies, the FDA will not recommend the use of ivermectin because, while the drug may help patients, such a recommendation would violate agency processes. Boxes need to be checked off in the right order. If a sponsor never shows up and the boxes aren’t checked off, the FDA’s standard approach is to tell Americans to stay away from the drug because it might be dangerous or ineffective. Sometimes the FDA is especially adamant, and its warnings are, frankly, alarming — as they were with ivermectin. Guilty until proven innocent. By the way, the FDA doesn’t always follow this procedure. Ironically, in the same warning it gave about using ivermectin, it recommended using facemasks to ward off COVID, even though the efficacy of masks is questioned. (See “How Effective Are Cloth Face Masks?” Winter 2021–2022.)

Drug Companies Want New Winners

To some, the fact that Merck, the originator and current marketer of ivermectin, would not say that it works against COVID is proof enough that it doesn’t. If a drug company wants to make money by selling drugs, they reason, the only explanation for Merck’s pessimistic statement is that the drug is ineffective. 

But Merck had two good reasons to discount ivermectin’s use against COVID and even outright warn against it. 

The first reason has to do with off‐label promotion. Once drugs are marketed, doctors can prescribe them for uses not specifically approved by the FDA. Such usage is called “off‐label.” Using ivermectin for COVID-19 is considered off‐label because that use is not specifically listed on the drug’s FDA‐approved label. 

While off‐label prescribing is widespread and completely legal, it is illegal for a pharmaceutical company to promote that use. If Merck said something as innocuous as suggesting that some clinical data favored ivermectin use against COVID, the drugmaker would set itself up for substantial penalties. During a particularly vigorous two‐year period, the Justice Department collected over $6 billion from drug companies for off‐label promotion cases. Merck’s lawyers haven’t forgotten that lesson. 

The second reason for Merck to discount ivermectin’s efficacy against COVID has to do with marketing strategy. Ivermectin is an old, cheap, off‐patent drug. Merck will never make much money from its sales. Drug companies aren’t looking to spruce up old products; they want new winners with the exclusivity that comes from long patent lives and the concomitant potential for higher prices. Not coincidentally, Merck recently released the clinical results for molnupiravir, and analysts are predicting multi‐billion‐dollar sales for the drug. 

While we can all be happy that Merck has developed a new therapeutic for COVID, we should realize that the FDA’s rules give companies an incentive to focus on new drugs while ignoring old ones. Ivermectin may or may not be a miracle drug for COVID, but both the FDA and Merck have incentives to refrain from announcing that it might be.

Statistical Significance

Researchers say that a result is statistically significant if they decide there is a true difference between the treatment group and the control group. Statisticians often use a 95% confidence level for statistical significance, meaning that the result in the treatment group is different enough from the control group that there is only one chance in 20 that the difference is the product of random chance alone. In the case of pharmaceuticals, if the clinical results are good and the results are statistically significant, the drug is deemed to be effective. If the results aren’t good or the results aren’t statistically significant with 95% confidence, the drug is deemed to not work. 

Consider one COVID patient outcome: the need for invasive ventilation. In a randomized, double‐blind, placebo‐controlled clinical trial by Ranjini Ravikirti et al., of 55 patients in the ivermectin arm, only one patient needed invasive ventilation while five in the placebo group of 57 did. In other words, it appears that ivermectin reduced the need for ventilators by 80%. Yet, the study’s authors concluded, “This study did not find any benefit with the use of ivermectin in … the use of invasive ventilation in mild and moderate COVID-19.” 

But one can reasonably conclude that the authors did find a benefit. A close look at their data shows 91.2% confidence that there was a difference. Because the authors used the 95% threshold, they stated that they had found no benefit. 

Similarly, an observational controlled trial of 288 patients found that treatment with ivermectin allowed twice as many patients to improve and get off mechanical ventilators (36.1% vs 15.4%). But authors Juliana Cepelowicz Rajter et al. report no benefit to ivermectin because they were “only” 93% confident of the difference. 

In both of these cases, the trials were quite small. That made it especially difficult to eclipse the 95% threshold. Yet, it’s incorrect to conclude, as the authors report, that the drug itself failed; rather, the desired result (barely) failed to eclipse the confidence level they were using. Note that drugs with deep‐pocketed sponsors typically don’t have this problem because the trials are usually big enough to reach statistical significance. 

Importantly, a meta‐analysis of all 12 of the above‐mentioned ventilation studies, encompassing a combined 2,316 patients, does produce a result that eclipses the 95% confidence level — it actually reaches 99.91%. Among all the patients, 30 out of 1,230 patients on ivermectin needed ventilators while 51 of 1,086 patients on placebo or standard therapies did. That suggests that ivermectin reduced the risk by 48%. The 95% confidence interval for the result is 38% to 78%, meaning there’s a 2.5% chance that the real benefit of ivermectin is less than 38% and a 2.5% chance that the real benefit is greater than 78%. In other words, it’s highly likely that ivermectin benefits patients hoping to avoid mechanical ventilators. 

To appreciate what it means to demand a 95% confidence level for each individual study, consider a hypothetical pharmaceutical company that has five investigators collecting separate clinical results on new Drug X. If the investigators publish their results individually and each uses a 95% confidence level, each investigator will find his or her results are not statistically significant. They would each conclude that the drug does not work. But suppose they pool their research and the resulting data set yields a positive result with 95% confidence; they would then say that Drug X does indeed work. Obviously, the authors’ choice of whether to combine their data or not does not change the drug’s efficacy. 

The problem of statistical significance is ubiquitous, and scientists get tripped up by it routinely. This is a problem that some are trying to address. In a March 2019 comment in the journal Nature, University of Basel zoology professor Valentin Amrhein et al. wrote, “The misuse of statistical significance has done much harm to the scientific community and those who rely on scientific advice.” The writers quickly gathered 800 signatories in their call to abandon using statistical significance to make dichotomous conclusions like “Drug X works” or “Drug Y doesn’t work.” Instead, they recommend using point estimates and confidence intervals, as we did above when discussing the effect of ivermectin on the need for ventilators.

Kooky Ivermectin Advocates

As noted above, Alexander concluded that the clinical evidence, while short of being definitive, suggests that ivermectin reduces COVID mortality by 40% when administered early, which just barely eclipses the standard statistically significant level at p = 0.04 (that is, 96% confidence that the results are not due to luck). But then Alexander backtracks and decides not to endorse using the drug to treat COVID. 

In doing this, he displays a prejudice, equating ivermectin support with being kooky. “I think the conventional wisdom — that the most extreme ivermectin supporters were mostly gullible rubes who were bamboozled by pseudoscience — was basically accurate,” he writes. To be sure, some of the drug’s advocates are unreliable, but their existence does not void the empirical evidence. Yet, for Alexander, because some people he doesn’t respect advocate ivermectin, he is hesitant to advocate the drug lest he be tainted by association. 

He further acknowledges that “if you say anything in favor of ivermectin, you will be cast out of civilization and thrown into the circle of social hell reserved for Klan members and 1/6 insurrectionists.” Not wanting to be relegated to this group of undesirables, he withholds his recommendation of ivermectin. In short, the scientific evidence led him to a tentative conclusion that he does not want to embrace because of social desirability bias. What happened to “follow the science?”

Pandemics Require Off‐Label Therapies

When the pandemic hit, the only hope for quick treatments resided in off‐label uses of currently available drugs. New therapeutics typically take over a decade to develop, precluding their use during an emerging pandemic. Therefore, people had to look for drugs that had already been marketed for other conditions and see if they held any promise for patients infected with SARS‐CoV‑2. Logically, because no drugs had been previously approved for use against this particular virus, our only hope was to repurpose old drugs for this new, off‐label use. Ivermectin is one such repurposed old drug. 

We are very fortunate that, in the past two years, we have found quality vaccines and now even therapeutics for this disease. But why exclude older drugs? In richer countries, people will benefit from a range of treatment options. And, in much of the world where the new drugs are scarce, people could use preventatives and treatments that are already available. Ivermectin could help address each of those demands. 

As described above, many medical authorities have claimed the drug does not work against COVID-19. Their reasons for claiming this may have more to do with biases and structural limitations than with the drug itself. Science has taken a back seat to prejudice and process. People are dying because many medical authorities say that therapies such as ivermectin do not work, while the actual clinical results suggest otherwise. These medical authorities should “follow the science” rather than rationalize their reasons not to.

Readings

• “Ivermectin: Much More than You Wanted to Know,” by Scott Alexander. Astral Codex Ten (blog), Nov. 17, 2021.
• “Ivermectin as a Potential Treatment for Mild to Moderate COVID–19: A Double Blind Randomized Placebo‐Controlled Trial,” by Ranjini Roy Ravikirti, Chandrima Pattadar, Rishav Raj, et al. MedRxiv, Jan. 9, 2021.
• “Scientists Rise Up Against Statistical Significance,” by Valentin Amrhein, Sander Greenland, and Blake McShane. Nature, March 20, 2019.
• “Use of Ivermectin Is Associated with Lower Mortality in Hospitalized Patients with Coronavirus Disease: The Ivermectin in COVID Nineteen Study,” by Juliana Cepelowicz Rajter, Michael S. Sherman, Naaz Fatteh, et al. Chest 159(1): 85–92 (2021).