Retracted COVID-19 studies expose holes in vetting of data firms

The scramble to research the novel coronavirus has exposed weaknesses in the vetting of healthcare data being supplied by a growing number of U.S. firms, a flaw that forced two of the most respected medical journals to pull studies last week.

The Lancet and the New England Journal of Medicine (NEJM) retracted COVID-19 studies over questionable patient health data supplied by a small company called Surgisphere.

U.S. researchers say they are routinely peppered with pitches from similar firms, with no widespread standard of how to verify their datasets.

For latest updates on coronavirus outbreak, click here

“Whenever you don't know where data comes from, and precisely how it was managed, then the number of incorrect conclusions you can draw is large,” said Kenneth Mandl, director of the Computational Health Informatics Program at Boston Children’s Hospital. “The field is young enough that those best practices may not be well established across the scientific community.”

The Lancet study, which concluded treating COVID-19 with hydroxychloroquine increased the risk of death, drew scrutiny from scientists worldwide. They suggested that patient outcomes data from several countries did not add up.

Mandeep Mehra, a Harvard Medical School professor and researcher with Brigham and Women’s Hospital in Boston, and his co-authors retracted the Lancet study and a second study in NEJM, saying they could not vouch for the veracity of the data and that Surgisphere would not provide its underlying data to independent auditors.

The retractions “are great examples of why science needs more of a ‘In God We Trust, everyone else needs to show their data’ approach,” said Ivan Oransky, vice president of editorial at Medscape and co-founder of the Retraction Watch blog.

Over the years, more than 1,500 studies have made it into Retraction Watch’s database because of data concerns such as data falsification by the authors, Oransky said. There have been about 15 medical paper retractions related to COVID-19, according to the website.

For years, U.S. healthcare data was concentrated in medical claims compiled by large insurers such as UnitedHealth Group Inc and the government. As hospitals have moved to electronic health records, independent analytics firms are buying hospital data sold without patient names.

The U.S. Food and Drug Administration now allows data gathered outside of clinical trials to be considered in its reviews of new drugs, providing another boost to the healthcare analytics industry.

'REVIEWING OUR PROCEDURES'

Coronavirus Worldometer | 15 countries with the highest number of cases, deaths due to the COVID-19 pandemic

It is not clear how Surgisphere accessed the data used in the two studies. Surgisphere did not return requests for comment.

At a time when researchers are scrambling to find a cure for COVID-19, data sets like those used in the now retracted studies can appear to be a “goldmine,” Mandl said.

A spokesman for Lancet publisher Elsevier said it will re-assess about 20 additional published articles that contain Surgisphere data.

A spokesman for Brigham and Women's Hospital said its data oversight mechanisms “were not applicable” to the Lancet and NEJM studies, adding it only provides “support, guidance and oversight on all agreements that involve the use of our institutional resources, including our patient data.”

Harvard Medical School would not comment on its role in vetting the data. Harvard and the Brigham declined to say whether further steps would be taken regarding the researchers’ handling of the matter.

While the NEJM and Lancet articles had external peer review, they relied on the authors to vouch for the data.

“We are reviewing our procedures, including how we assess research analyzing large datasets based on electronic medical record data,” said NEJM spokeswoman Jennifer Zeis.

After the retraction, the Lancet said it was conducting a review, and that serious scientific questions had been raised.

Dr. Deneen Vojta, executive vice president of UnitedHealthcare’s global research and development division, said it is important to ask questions about how a group obtained their data, and that it must be made available for scrutiny.

Dr. Peter Bach, from Memorial Sloan Kettering in New York, said his research team regularly assesses data offered by these new firms to make sure it jibes with its own knowledge, such as ages of people having a certain surgery or the number of prescriptions the firm says were dispensed.

"We do lots of tire kicking,” Bach said.