Analysis
Data were analyzed to address the 3 research questions for the process evaluation utilizing the Logic Model as the conceptual framework (Figure). To determine how often various intervention components were delivered, the total hours spent at each practice and the total number of contacts with each practice by intervention component were calculated from the PF activity sheets.
To determine intervention quality, triangulation31 was used to attain a complete understanding of the quality of implementation. Multiple data sources and analysis methods were used to reveal the underlying dimensions of quality. All data sources were reviewed and analyses were conducted independently by 2 members of the investigation team. The members of the team held a debriefing session to discuss their findings and seek consensus. First, the monthly narrative reports across intervention sites were summarized to qualitatively describe the type, breadth, and scope of activity for each intervention component. Second, the activity descriptions and open-ended interview responses were content analyzed32 and coded, and frequencies were generated. The goal of this analysis was to identify significant descriptions of which intervention elements worked well and which did not. Finally, intervention and control practices were compared with contingency tables, and a chi-square statistic was used to determine differences on questionnaire responses concerning practice changes over the period of the intervention.
To determine physician satisfaction, open-ended satisfaction survey responses were coded and frequencies generated for ratings of overall satisfaction with the performance of the PF and the intervention.
Results
PF Program Implementation
Table 3 shows the number of hours spent on project activities during the period of the intervention. The PFs spent the largest proportion of their time (28%) on administrative duties, such as team meetings, telephone calls, internal reporting, preparing the project newsletter, coordinating networking conferences for intervention practices, photocopying, and filing. Sixteen percent of the PFs’ time was spent on-site facilitating changes to improve preventive care in the practice. Travel accounted for an average of 12% of the PFs’ time, although this varied depending on the distance to the practices.
Table 4 provides information on the number of contacts and hours spent on-site for each component of the intervention. On average, each intervention practice was contacted 33 times by a PF, with each visit lasting an average of 1 hour and 45 minutes. The most frequent forms of contact concerned developing reminder systems, conducting chart audits and providing feedback preventive care performance, and working to achieve consensus on the adoption of preventive care strategies. Both academic detailing to physicians and supplying critically appraised patient education materials averaged approximately 20 minutes but involved a great deal of preparation time. Few practices were interested in posters in the waiting room or a patient newsletter on prevention, so fewer contacts were made for those components.
Quality of Implementation
To assess quality, the frequency of each component of the intervention was tallied, physician feedback on the usefulness of intervention components was summarized, and self-reported practice changes between intervention and control physicians was reported.
Intervention Scope
Audit and Feedback. All 22 intervention practices received a presentation by the PF on the initial audit results to raise awareness of preventive care practice patterns. This was usually done in a kick-off meeting involving both physicians and nurses and often required more than one presentation to cover the various staff in the practice. Twenty practices requested subsequent analyses of data to follow their rates of performance. In addition, 18 practices requested audits of their charts for specific maneuvers, such as influenza vaccination and mammography
Consensus Building. All practices were involved in meetings with the PF to identify opportunities for improvement, assess needs, and select priority areas and strategies for improving preventive care performance. Interviews were conducted with nurses and other staff to promote their role in preventive care delivery.
Academic Detailing. Twenty-one out of 22 sites agreed to receive and discuss critically appraised evidence for the preventive maneuvers under study, and some requested similar information on other preventive areas, such as cholesterol and osteoporosis.
Reminder Systems. All of the intervention sites implemented some form of reminder system. Eighteen sites implemented a preventive care flow sheet; 2 sites used a chart stamp; and 2 sites implemented a computerized reminder system. Nineteen sites developed recall initiatives for flu vaccine, mammography, and Papanicolaou tests. Seventeen sites implemented chart stickers for smoking counseling or mammography.
Opinion Leaders. All sites received copies of the PF project newsletter that contained articles by influential individuals describing the importance of preventive care and descriptions of colleagues’ preventive care implementation efforts. Most practices attended a workshop that included an influential keynote speaker, and 27% of the participating physicians shared their knowledge about preventive care through publishing in the newsletter and/or public speaking.