POINT: SCIP is both efficacious and effective.
The Surgical Care Improvement Project (SCIP) was a national campaign that set out to reduce surgical mortality and morbidity by 25% by 2010 through recommendations in targeted areas: wound infections, perioperative MIs, and venous thromboembolism. The recommendations have become pay-for-performance measures. There are seven in the area of infectious disease to reduce surgical site infections. There is one measure for reducing perioperative MI: Continue beta-blockers (for patients who are on them) in the perioperative period. For venous thromboembolism prevention, give prophylaxis within 24 hours before to 24 hours after surgery.
It’s key to understand the difference between efficacy and effectiveness. I think we would all agree that the SCIP measures have efficacy. Efficacy trials determine whether an intervention produces the expected result under ideal circumstances. Effectiveness trials measure the degree of beneficial effect under "real world" clinical conditions. The problem with effectiveness trials is that those real-world conditions may change the effect, or they might just change the ability to measure the effect.
Dr. Robert S. Lagasse
I believe that the SCIP measures have proven efficacy because they all are based upon randomized controlled trials that were identified by systematic reviews amenable to meta-analysis. All of these measures are Level 1 recommendations, based on the highest forms of evidence. The studies that Dr. Barash uses to criticize SCIP measures are cohort studies. They do not randomize. There may be unknown confounding variables.
There have been effectiveness trials that show that the SCIP measures do work. One showed a 27% decrease in surgical site infections, another showed a 62% decrease in surgical site infections, and a third showed a 39% decrease in surgical site infections.
Perhaps the strongest endorsement of efficacy of the SCIP measures comes from Dr. Kaveh G. Shojania, who has written several reviews of the efficacy of medical interventions. This guy is like Mikey from the old Life cereal commercials ... he hates everything. He said there were 11 patient safety practices rated most highly in terms of strength of the evidence, and 3 are SCIP measures: appropriate use of prophylaxis to prevent venous thromboembolism in patients at risk, use of perioperative beta-blockers in appropriate patients, and appropriate use of antibiotic prophylaxis in surgical patients (AHRQ Publication No. 01-E058).
Several trials published by pretty good researchers in reputable journals show a lack of effectiveness of SCIP measures. Even those researchers admit to the efficacy of SCIP measures. The lead investigator of the best effectiveness trial, a retrospective cohort study, wrote, "There are several explanations as to why we did not observe an association between timely antibiotic administration and surgical site infection (SSI). The first is that timely antibiotic administration does not diminish SSI risk. This is an unlikely interpretation. There are numerous randomized controlled trials and observational studies that demonstrate the efficacy of prophylactic antibiotics in reducing SSI for various surgical procedures" (Ann. Surg. 2011;254:494-9).
A separate retrospective cohort study showed a decrease in surgical site infection only if two or more SCIP recommendations were followed (JAMA 2010;303:2479-85). Shocking – if you give the wrong antibiotic at the right time, it might not work.
Another retrospective cohort study found no association with adjusted complications and SCIP compliance. Hospitals in the lowest compliance group had patients in lower-income ZIP codes and lower unadjusted complication rates. So, poor people go home and don’t come back, perhaps because of payment considerations. The study didn’t have enough patients; it also used measures that don’t apply to SCIP (Arch. Surg. 2010;145:999-1004).
SCIP did not design these measures for pay-for-performance programs. The intent was to decrease perioperative complications by 25% by 2010. When you start changing the baseline with pay-for-performance, it doesn’t work. In a study by Hawkins et al., the authors tested the hypothesis that documented compliance with antibiotic prophylaxis guidelines on a pediatric surgery service does not reflect adherence to guidelines as intended. In a 7-week observational study of elective pediatric surgical cases, adherence was evaluated for appropriate administration, type, timing, weight-based dosing, and redosing of antibiotics. Prophylactic antibiotics were administered appropriately in 141 of 143 cases (99%). Of 100 cases in which antibiotic prophylaxis was indicated, compliance was documented in 100% of cases in the electronic medical record; but only 48% of cases adhered to all five guidelines. Lack of adherence was due primarily to dosing or timing errors.
The SCIP measures, however, are based on best evidence. They are tightly linked with the desired outcomes. They are measurable and effectible, as demonstrated in multiple randomized controlled trials. Studies of effectiveness have had variable results due to methodological flaws.