Shot in the dark: Early Covid-19 vaccine efficacy explained

This week has seen a flurry of good news from Covid-19 vaccine developers, with Pfizer Inc. and BioNTech SE trumpeting early data indicating their mRNA candidate is more than 90% effective.

A Russian project came out a day later, touting 92% efficacy for the Sputnik V candidate, named after the Soviet-era satellite, based on a smaller data set.

How do manufacturers arrive at efficacy numbers?

In Pfizer's case, it waited until 94 volunteers in its late-stage clinical trial of more than 43,500 people - half got the vaccine, the other half got a placebo - tested positive after developing symptoms.

For 90%-plus efficacy, no more than eight people among those who tested positive had received the vaccine, with the rest having received the placebo.

"Roughly speaking, it's probably around eight to 86 cases in the treated and placebo groups," David Spiegelhalter, a Cambridge professor of risk and an expert in statistics, told Reuters.

"You don't need a lot of fancy statistical analysis to show that this is deeply impressive. It just hits you between the eyes."

In Russia, Sputnik V-developer Gamaleya Institute reached its preliminary 92% efficacy figure based on 20 illnesses in 16,000 volunteers as its late-stage trial progresses. It aims to reach 40,000 people.

Of the 16,000 people, about quarter got the placebo.

"It suggests that there is some effect, but it's insufficient to estimate the magnitude of it," Spiegelhalter said.

Read: Anthony Fauci stresses on need for equitable access of Covid-19 vaccines

How many people must get sick in big vaccine trials?

Some experts say that, ideally, 150 to 160 people in a trial of tens of thousands of participants must get sick before making a reliable assessment of a vaccine's efficacy. That's a bit of a rule of thumb, though, open to interpretation.

"There is no such regulatory standard requiring X number of events for making a reliable decision," the government-funded Swiss Clinical Trial Organisation said. "The amount of (infections) has to be seen in relation to the disease and its risk profile. It's rather a case-by-case evaluation."

Typically, regulators strive to have at least 95% certainty that the trial read-out is not the result of random variations with nothing to do with the tested compound.

For trial sponsors, there is safety in numbers as a large enough trial can ensure that 95% reliability hurdle is cleared.

But the larger the underlying clinical benefit, the fewer trial participants needed to create that clarity.

In Pfizer and BioNTech's trial, they planned a final analysis when 164 people had become sick, with multiple, pre-planned interim analyses along the way. They skipped an analysis at 32 patients, and once they were ready to release a look at the 62-person mark, 94 had come down sick.

Details from the Russian trial are unclear, without access to its protocol.

Read: WHO in talks with Russian institute on Sputnik V Covid-19 vaccine

How do these results stack up to other drugs or vaccines for other illnesses?

In normal drug trials, for diseases like terminal cancer, benefits of new medicines may be less apparent, with survival benefits of just a few months sometimes revolutionary for patients at death's door.

For vaccines, however, marginal protection is inadequate, and the World Health Organization ideally wants to see at least 70% efficacy in trials, while the U.S. Food and Drug Administration wants at least 50%.

The 90% efficacy reported in the Pfizer and Russian trials beats those and appears to exceed that of typical flu vaccines, which the U.S. Centers for Disease Control and Prevention (CDC) estimate reduce the risk of sickness by 40%-60%.

For other shots, the CDC estimates the efficacy of a two-shot measles vaccine at 97%, and a two-dose chickenpox vaccine at 90%. Two doses of polio vaccine are 90% effective, rising to nearly 100% with a third.

Read: Scientists develop novel method to preserve vaccines without refrigeration

Can we expect efficacy rates to hold up as trials advance?

Pfizer acknowledged on Monday that its final vaccine efficacy percentage may vary. Still, Spiegelhalter said the study's design seems likely to generally hold up, based on the 94 sick participants.

"In this case, the effect is so huge, even if there is a little bit of fallback - if the effects become slightly smaller over time - that is very unlikely to be significant."

What about real-world efficacy, should the vaccines be approved?

The interim data is promising, since it appears to demonstrate that a vaccine can be effective in preventing Covid-19.

The jump to mass vaccinations, however, presents new hurdles, in particular for an mRNA vaccine like Pfizer and BioNTech's that must be stored and shipped at minus 70 degrees Celsius (-94°F).

Moreover, the Pfizer-BioNTech vaccine requires two doses, ideally 21 days apart. If people do not stick to the timetable, it may affect the vaccine's efficacy.

Protection against the mumps, for instance, drops from nearly 90% to 78%, if people don't get a follow-up shot.

Swiss epidemiologist Marcel Tanner, president of Switzerland's Academies of Arts and Sciences and one of the government's top Covid-19 science advisers, expects possible variations in efficacy among older people, whose immune systems wane with time, or those with immune disorders.

"Efficacy says, 'Does it work?' Effectiveness says, 'Can it be applied? Can you carry the efficacy to the people?'" Tanner said. "But no question: 90% efficacy, at that stage, is a pretty good result."