An Operative Performance Rating System for Urology Residents
An operative performance rating system (OPRS) for Urology residents was developedfor 6 sentinel urological procedures. This study tests the reliability, validity, and feasibility of an OPRS for Urology residents.
Materials and Methods
Each procedure OPRS contained a 3-point case difficulty scale, 4 to 6 procedure specific items, 3 general items, and an overall performance item. A Likert scale of 1 to 5 was used for each item. A single video/audio record of each procedure was evaluated by the faculty, and single-item inter-rater agreement was measured by comparison of the observed variance and random measurement-error variance.
Resident operative performance evaluations were completed on-line. Internal consistency reliability was measured using Cronbach's alpha. Overall scale scores by PGY level were compared using one way analysis of variance (ANOVA).
Faculty evaluation of video/audio records showed an inter-rater agreement range of 0.71 to 0.92. Faculty evaluations of resident operative performance showed an internal consistency reliability range of 0.91 to 0.95. Significant differences in overall scale scores between PGY levels were demonstrated for 3 of the 6 procedures (p <0.0016).
An OPRS for Urology residents is feasible using an internet based resident management system. Inter-rater agreement and internal consistency reliability meet threshold limits for checklist evaluation instruments. The OPRS can discriminate PGY level of resident training. A validated OPRS can offer residents immediate and objective feedback on surgical performance, and enable program directors to monitor progress in resident operative performance.
1Department of Surgery, Division of Urology Southern Illinois University School of Medicine, Springfield, IL
2Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL
3Cancer Care Specialists of Central Illinois, S.C., Decatur, IL
Sentinel Procedure OPRS Forms
- Elective Scrotal Surgery
- Transurethral Resection of Bladder Tumor
- Ureteroscopic Removal of Distal Ureteral Calculus
- Cystoscopy, Retrograde Pyelography, and Placement of Internal Ureteral Stent
- Transrectal Ultrasound Guided Biopsy of the Prostate
- Transurethral Resection of Prostate
(The manuscript for this study has been accepted for publication in The Journal of Urology with anticipated publication date of November 2012. Above are links to example OPRS forms developed for each of the six sentinel procedures.)
Aaron Benson, M.D.,1 Stephen Markwell, M.A.,2 Tobias Köhler, M.D., M.P.H.,1 Thomas H. Tarter, M.D., Ph.D.3