Intervention
Resident physicians were randomly assigned to a treatment group that was exposed to the CART (n=15) and a control group that used existing blank history and physical examination forms (n=16). Resident physicians from both groups precepted patients with the same faculty physicians on a regular basis. Physicians in the control group used a different area of the clinic facility from the treatment group to avoid cross-use of the CART form. The 3 time periods of the study were: pre-intervention phase (3 months), intervention phase (6 months), and post-intervention phase (3 months). The pre-intervention phase allowed for the establishment of a baseline measurement of preventive medicine practiced by physicians in both groups. In the intervention phase, the CART forms were placed on all charts of new patients seen by physicians in the experimental group. The forms were not introduced by any formal instruction. No preventive medicine lectures were given to physicians in either group during the course of the study. Finally, in the post-intervention phase the CART was no longer placed on charts.
Measures
Blinded chart reviews were performed by the principal investigator and 2 other independent reviewers on randomly selected new patients. The physicians were expected to complete a history and physical examination on each new clinic patient within the first 3 visits. Chart reviews occurred before use, during use, and after removal of the CART. They evaluated charts for screening appropriateness (not the absolute frequency of interventions). No credit was given when a preventive intervention was performed that was not indicated. A total of 300 charts were reviewed for the treatment group and 308 for the control group. The inter-rater agreement between the principal investigator and 2 other independent reviewers ranged between 93% and 98%. The median kappa statistic among 16 screening recommendations revealed very good agreement at 0.81 (all P <.042). Kappa could not be calculated for 4 of the screening recommendations, because the observed agreement of the 3 raters (100%) was equal to the expected agreement for those recommendations. All physicians were given a pretest measuring knowledge of the USPSTF recommendations at the beginning of the pre-intervention phase and a posttest near the end of the post-intervention phase. Finally, as physician learning and behavior were the focus of this study, patient adherence was not measured.
Data Analysis
Recommendations were organized into 4 categories: history, examination and laboratory, counseling, and prophylaxis. Tests of significance were calculated using a chi-square test that is designed to compare proportions among many independent samples.21 To control for statistical error, we used a Bonferroni adjusted P value. Because 20 screening recommendations were analyzed, a P value of .0025 (.05/20) was used as the cutoff for statistical significance.
Scores from the knowledge test were analyzed using a mixed factorial analysis of variance. The between-groups factor was group (treatment or control) and the within-subjects factor was test period (pretest or posttest). This analysis was conducted to see if the tool would lead to a group-by-test period interaction. Given the repeated use of the CART, it was thought that the treatment group might show a greater improvement in test scores than the control group.
Results
Impact of the CART
The Table 1 shows the percentages of patients who were appropriately screened during each period for the treatment and control groups. Significant increases were observed for all recommendations in the treatment group, except for occult blood and tetanus and diptheria booster. The mean absolute increase in the percentage of appropriately screened patients was 45%, 21%, and 15% for the recommendations in the history, examination and laboratory, and counseling categories, respectively. Removal of the annotated reminder during the post-intervention period brought the percentages of appropriate screenings down to baseline levels. Increases during the intervention phase were statistically significant for 16 of the 20 recommendations.
In the control group, the percentages of appropriately screened patients remained relatively stable. Although some variation is visible across the 3 study periods, there are no consistent trends among the recommendations, and none of the chi-square tests reached the Bonferroni level of significance. The Figure 1 shows the median percentage of patients screened during the baseline, intervention, and post-intervention periods. The average percentage of appropriate screenings for the clinicians using the CART increased during the intervention period and then returned to baseline levels after the tool was removed from use. The inverted V pattern repeatedly observed for the treatment group is not seen with the control group.
Screening Knowledge Scores
Only the main effect for the test period reached statistical significance. The mean test scores were 54.66 and 62.35 for clinicians at the pretest and posttest periods, respectively (F [1,27] = 23.89; P <.0005). The main effect for group was not statistically significant. The mean test scores for the treatment and control groups were 58.59 and 58.13, respectively (F [1,27] = 0.27; P <.608). The group-by-test period interaction also was not statistically significant (F [1,27] = 1.42, P=.244). Thus, although physicians had higher posttest knowledge scores than pretest scores, this increase was similar for treatment and control groups.