No ‘good-housekeeping’ seal
For articles based on large, real-world datasets, the Lancet Group will now require that editors ensure that at least one peer reviewer is “knowledgeable about the details of the dataset being reported and can understand its strengths and limitations in relation to the question being addressed.”
For studies that use “very large datasets,” the editors are now required to ensure that, in addition to a statistical peer review, a review from an “expert in data science” is obtained. Reviewers will also be explicitly asked if they have “concerns about research integrity or publication ethics regarding the manuscript they are reviewing.”
Although these changes are encouraging, Harlan Krumholz, MD, professor of medicine (cardiology), Yale University, New Haven, Conn., is not convinced that they are realistic.
Dr. Krumholz, who is also the founder and director of the Yale New Haven Hospital Center for Outcome Research and Evaluation, said in an interview that “large, real-world datasets” are of two varieties. Datasets drawn from publicly available sources, such as Medicare or Medicaid health records, are utterly transparent.
By contrast, Surgisphere was a privately owned database, and “it is not unusual for privately owned databases to have proprietary data from multiple sources that the company may choose to keep confidential,” Dr. Krumholz said.
He noted that several large datasets are widely used for research purposes, such as IBM, Optum, and Komodo – a data analytics company that recently entered into partnership with a fourth company, PicnicHealth.
These companies receive deidentified electronic health records from health systems and insurers nationwide. Komodo boasts “real-time and longitudinal data on more than 325 million patients, representing more than 65 billion clinical encounters with 15 million new encounters added daily.”
“One has to raise an eyebrow – how were these data acquired? And, given that the U.S. has a population of around 328 million people, is it really plausible that a single company has health records of almost the entire U.S. population?” Dr. Krumholz commented. (A spokesperson for Komodo said in an interview that the company has records on 325 million U.S. patients.)
This is “an issue across the board with ‘real-world evidence,’ which is that it’s like the ‘Wild West’ – the transparencies of private databases are less than optimal and there are no common standards to help us move forward,” Dr. Krumholz said, noting that there is “no external authority overseeing, validating, or auditing these databases. In the end, we are trusting the companies.”
Although the Food and Drug Administration has laid out a framework for how real-world data and real-world evidence can be used to advance scientific research, the FDA does not oversee the databases.
“Thus, there is no ‘good housekeeping seal’ that a peer reviewer or author would be in a position to evaluate,” Dr. Krumholz said. “No journal can do an audit of these types of private databases, so ultimately, it boils down to trust.”
Nevertheless, there were red flags with Surgisphere, Dr. Rao pointed out. Unlike more established and widely used databases, the Surgisphere database had been catapulted from relative obscurity onto center stage, which should have given researchers pause.