Simulator and 2 tools: Validation of performance measures from a novel neurosurgery simulation model using the current Standards framework

Background Ventriculostomy is a common neurosurgical procedure with a relatively steep learning curve. A low-cost, high-fidelity simulator paired with procedure-specific performance measures would provide a safe environment to teach ventriculostomy procedural skills. The same validated simulation mo...

Full description

Saved in:
Bibliographic Details
Published in:Surgery 2016-09, Vol.160 (3), p.571-579
Main Authors: Rooney, Deborah M., PhD, Tai, Bruce L., PhD, Sagher, Oren, MD, Shih, Albert J., PhD, Wilkinson, David Andrew, MD, Savastano, Luis E., MD
Format: Article
Language:English
Subjects:
Clinical Competence
Humans
Internship and Residency
Models, Anatomic
Simulation Training
Surgery
Ventriculostomy - education
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Background Ventriculostomy is a common neurosurgical procedure with a relatively steep learning curve. A low-cost, high-fidelity simulator paired with procedure-specific performance measures would provide a safe environment to teach ventriculostomy procedural skills. The same validated simulation model could also allow for assessment of trainees' proficiencies with measures that align with Accreditation Council for Graduate Medical Education milestones. This study extends previous work to evaluate validity evidence from the simulator, its newly developed performance assessment, the Ventricolostomy Procedural Assessment Tool, and the Objective Structured Assessment for Technical Skills. Methods After Institutional Review Board exemption, performance data were collected from 11 novice and 3 expert neurosurgeons ( n  = 14). Participants self-reported their ability to perform tasks on the simulator using the Ventricolostomy Procedural Assessment Tool, an 11-item, step-wise instrument with 5-point rating scales ranging from 1 (unable to perform) to 5 (performs easily and smoothly). De-identified operative performances were videotaped and independently rated by 3 neurosurgeons, using the Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills. We evaluated multiple sources of validity evidence (2014 Standards) to examine psychometric quality of the measures and to test our assumption that the tools could discriminate between novice and expert performances adequately. We used a multifacet Rasch model and traditional indices, such as Cronbach alpha, intraclass correlation, and Wilcoxon signed-rank test estimates. Results Validity evidence relevant to test content and response processes was supported adequately. Evidence of internal structure was supported by high interitem consistency ( n  = 0.95) and inter-rater agreement for most Ventricolostomy Procedural Assessment Tool items (Intraclass correlation coefficient = [0.00, 0.91]) and all Objective Structured Assessment for Technical Skills items (Intraclass correlation coefficient = [0.80, 0.93]). Overall, novices performed at a lower level than experts on both scales ( P  
ISSN:0039-6060
1532-7361
DOI:10.1016/j.surg.2016.03.035