No single standardized assessment of medical student neurology clerkships provides a complete picture of the student's knowledge, skills, and professionalism, according to an evaluation conducted at the Harvard Medical School, Boston.
Rather, it is a composite approach which relies on results from three different assessments that ultimately offers medical students the most “transparent and fair” evaluation possible, and can most accurately take into account the different strengths of each individual student, according to the evaluation.
The research for this study was supported in part by the Academy at Harvard Medical School and also by the Birmingham Foundation.
The authors reported no conflicts of interest in relation to this study.
The study was prompted because “There's always the concern that the student who brings the better doughnuts gets the better grade,” said Dr. Jeremy Schmahmann, who is the the lead author of the study and also the neurology clerkship director at Massachusetts General Hospital.
However, by using a composite of three separate evaluations, students can have confidence in the final grade for the rotation—regardless of whether they bring doughnuts. And that final grade often can go on to influence their final choice of medical specialty, he said.
Researchers from Massachusetts General Hospital and the Harvard Medical School compared the results of the Subjective Evaluation Form (SEF), the Bedside Examination Exercise (BEE), and also the National Board of Medical Examiners Shelf examination (Shelf) in grading those medical students who were, at that time, participating in Harvard Medical School's 1-month-long neurology clerkship.
The researchers compared the final grades of 71 students who were assessed with the Subjective Evaluation Form alone and 409 students using the Bedside Examination Exercise alone from 1991 to 2002.
They also performed a prospective study of 132 students who were assessed using all three tests—the BEE, SEF, and the Shelf—between 2003 and 2006.
Among the 132 students who were tested on all three of the instruments, the researchers found that there was a normal distribution of scores across the class for each test.
However, the researchers found poor agreement on knowledge between the three tests.
For example, the comparison showed that the faculty members and residents who worked with students in the wards typically tended to overrate student knowledge, compared with the results of the Shelf examination.
“None of the test instruments, no matter how fairly administered we thought they were, was sufficiently reliable or valid to serve as the final grade for the rotation,” the researchers wrote.
As a result, the researchers developed a composite score that is based on weighted versions of the three assessments. The makeup of the composite exam is 70% of the Subjective Evaluation Form, 15% of the Bedside Examination Exercise, and 15% of the National Board of Medical Examiners Shelf examination.
In applying the composite process, students would be required to pass all three tests to graduate from the rotation.
The composite score had a normative distribution among all of the 132 students who were tested on it (Neurology 2008;70:706–12).
The study demonstrates what clerkship directors already know very well, and that is that no single instrument is capable of measuring everything, said Dr. Ralph Jozefowicz.
Dr. Jozefowicz is the neurology clerkship director at the University of Rochester (N.Y.) and also the chair of the education committee for the American Academy of Neurology.
“Each of these tools examines different things,” said Dr. Jozefowicz, who reviewed the article before publication.
The study drives home the point that using only the traditional clerkship evaluation—the Subjective Evaluation Form—is insufficient, he said.
If the evaluation is fair and comprehensive, taking into account all of the aspects of what makes a good clinician, it will ensure that students will strive to excel in all areas, he said.
“Testing drives the curriculum. People study what you test them on,” Dr. Jozefowicz said.
But there is at least one significant barrier to having more neurology programs around the country adopt the combined approach.
Programs must pay to use the Shelf exam.
Additionally, the Bedside Examination Exercise carries a monetary cost as well as considerable faculty time investment, said Jozefowicz.