SIU Division of Urology 
Phone: 217-545-8000
Fax: 217-545-7305 
Email: urology@siumed.edu  
Make an Appointment  

An Operative Performance Rating System for Urology Residents

Purpose

An operative performance rating system (OPRS) for Urology residents was developed for 6 sentinel urological procedures. This study tests the reliability, validity, and feasibility of an OPRS for Urology residents.

Materials and Methods

Each procedure OPRS contained a 3-point case difficulty scale, 4 to 6 procedure specific items, 3 general items, and an overall performance item. A Likert scale of 1 to 5 was used for each item. A single video/audio record of each procedure was evaluated by the faculty, and single-item inter-rater agreement was measured by comparison of the observed variance and random measurement-error variance.

Resident operative performance evaluations were completed on-line. Internal consistency reliability was measured using Cronbach's alpha. Overall scale scores by PGY level were compared using one way analysis of variance (ANOVA).

Results

Faculty evaluation of video/audio records showed an inter-rater agreement range of 0.71 to 0.92. Faculty evaluations of resident operative performance showed an internal consistency reliability range of 0.91 to 0.95. Significant differences in overall scale scores between PGY levels were demonstrated for 3 of the 6 procedures (p <0.0016).

Conclusions

An OPRS for Urology residents is feasible using an internet based resident management system. Inter-rater agreement and internal consistency reliability meet threshold limits for checklist evaluation instruments. The OPRS can discriminate PGY level of resident training. A validated OPRS can offer residents immediate and objective feedback on surgical performance, and enable program directors to monitor progress in resident operative performance.

1 Department of Surgery, Division of Urology Southern Illinois University School of Medicine, Springfield, IL

2 Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL

3 Cancer Care Specialists of Central Illinois, S.C., Decatur, IL

Sentinel Procedure OPRS Forms

 

Aaron Benson, M.D.,1 Stephen Markwell, M.A.,2 Tobias Köhler, M.D., M.P.H.,1 Thomas H. Tarter, M.D., Ph.D.3