Deliberate error-seeding experiments involving a large congenital heart surgery database showed that even small levels of miscoding can substantially change mortality estimates. This was especially true for miscoding of procedure type and for operations with mortality below 10%.
Such error-driven variations in mortality estimates are especially troubling in an era when registry databases are more and more expected to form the foundation of risk analysis for various operations, and might even be used to evaluate the doctors and the institutions that perform these operations. In addition, errors in databases are not uncommon. One recent study of a carefully audited California database reported at least one diagnostically relevant error in 63% of patient records, according to Dr. Steve Gallivan and his colleagues.
Computer simulation techniques were used to create realistic analysis scenarios based on data from the Toronto Cardiovascular Surgery Database for Congenital Heart Surgery, which contains information on nearly 18,000 operations. This includes outcomes for 132 operation types from which 30 marker operations were chosen, each of which had been reported at least 100 times in the database and had nonzero mortality.
Four thought experiments were performed using the data on the marker operations. In the first experiment, the only errors introduced were random miscoding of outcomes with three scenarios: error rates of 1%, 3%, and 5%. Each of these scenarios showed considerable changes in mortality rates, especially when the true mortality rate was small (Eur. J. Cardiothorac. Surg. 2008;33:334–40).
In the second experiment, the only errors introduced comprised random omission of data at rates of 0%, 10%, or 20%, with the miscoding of outcomes fixed at 1%; these scenarios showed that random omission of data had no discernible effect on inaccuracies in mortality rate estimates. This was predicted by mathematical modeling.
In the third thought experiment, errors introduced comprised random outcome miscoding at different rates for deaths and survivors. A progressive increase in estimation error was seen when mortality rates fell, “and the scale of such overestimation is alarming for mortality rates below 10%,” according to Dr. Gallivan of University College, London, and his international colleagues.
The final thought experiment regarded introduced errors from miscoding of the operation type with no data omission or outcome miscoding. Three operations illustrated the potential dangers of such errors: ASD/secundum repair (recorded mortality rate 0.2%), TGA repair/arterial switch (mortality rate 9.0%), and the Norwood operation (mortality rate 36.3%). The assumption was made that each operation had an equal probability of being miscoded as one of the other two. As predicted from mathematical modeling, as the miscoding rate increased, the gross mortality rate for ASD/secundum repair became increasingly overestimated, the rate for TGA repair remained relatively the same, and the rate for Norwoods became increasingly underestimated, according to the authors.
“The results reported here sound a loud note of caution and perhaps it is time for a reappraisal of the clinical database structure. … There is often a somewhat misplaced belief that if one gathers a lot of data, then, if analyzed cleverly enough, they will reveal a new truth. … This view is wrongheaded; the reality is that the more data items that are collected, the more errors occur,” the authors stated. “The results we describe are alarming. Even moderate levels of error can lead to substantial inaccuracy in estimates of mortality rates and in some circumstances these inaccuracies can be gross, especially at the low mortality rates that are now prevalent in cardiothoracic surgery,” they added. “Even with … labor intensive methods, it is unlikely that errors will be completely eradicated. In view of this it is perhaps wise to adopt a more skeptical attitude to quantitative results, especially in relation to rates that are small.”
“Any data collection which is not verified in a professional way is not valid and can be misused and can be used against our profession. That's why I think that verification of data at the various levels, including the local institutional level, including the automatic and computerized level, up to visiting the sites and checking 100% of the data, is of the greatest importance,” said Dr. B. Maruszewski, one of the physicians responsible for the European Association for Cardio-Thoracic Surgery Congenital Database, in comments delivered at the paper's original presentation at the 2007 annual meeting of the European Association for Cardio-Thoracic Surgery.
As part of that discussion, Dr. Jaroslav Stark, one of the paper's authors, added, “You should collect as few data as possible, but even if your data set is only 20–25 items, it is still very important to check, because, as we have shown, a small error of 1% can increase your mortality estimates by five times.”