Operative Performance Rating System
Purpose
An operative performance rating system (OPRS) for Urology residents was developed for 6 sentinel urological procedures. This study tests the reliability, validity, and feasibility of an OPRS for Urology residents.
Materials and Methods
Each procedure OPRS contained a 3-point case difficulty scale, 4 to 6 procedure specific items, 3 general items, and an overall performance item. A Likert scale of 1 to 5 was used for each item. A single video/audio record of each procedure was evaluated by the faculty, and single-item inter-rater agreement was measured by comparison of the observed variance and random measurement-error variance.
Resident operative performance evaluations were completed on-line. Internal consistency reliability was measured using Cronbach's alpha. Overall scale scores by PGY level were compared using one way analysis of variance (ANOVA).
Results
Faculty evaluation of video/audio records showed an inter-rater agreement range of 0.71 to 0.92. Faculty evaluations of resident operative performance showed an internal consistency reliability range of 0.91 to 0.95. Significant differences in overall scale scores between PGY levels were demonstrated for 3 of the 6 procedures (p <0.0016).
Conclusions
An OPRS for Urology residents is feasible using an internet based resident management system. Inter-rater agreement and internal consistency reliability meet threshold limits for checklist evaluation instruments. The OPRS can discriminate PGY level of resident training. A validated OPRS can offer residents immediate and objective feedback on surgical performance, and enable program directors to monitor progress in resident operative performance.
1 Department of Surgery, Division of Urology Southern Illinois University School of Medicine, Springfield, IL
2 Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL
3 Cancer Care Specialists of Central Illinois, S.C., Decatur, IL
Sentinel Procedure OPRS Forms
- Elective Scrotal Surgery
- Transurethral Resection of Bladder Tumor
- Ureteroscopic Removal of Distal Ureteral Calculus
- Cystoscopy, Retrograde Pyelography, and Placement of Internal Ureteral Stent
- Transrectal Ultrasound Guided Biopsy of the Prostate
- Transurethral Resection of Prostate
Aaron Benson, M.D.,1 Stephen Markwell, M.A.,2 Tobias Köhler, M.D., M.P.H.,1 Thomas H. Tarter, M.D., Ph.D.3