AI in Assessment
AI in Assessment
Ethics, Innovation and Research
The use of Artificial Intelligence (AI), including Natural Language Processing (NLP), in medical education and assessment is ever evolving, and the technological advances and capabilities it enables have the potential to transform assessment development and scoring.
In March 2024, NBME researchers Victoria Yaneva, PhD, Manager, NLP Research, and Kimberly Swygert, PhD, Director, Test Development Innovations, sat down with Dr. Andrea Anderson, MD, Associate Chief of the Div. of Family Medicine, The George Washington University School of Medicine and Health Sciences, to discuss:
-
the critical role human oversight must play in the incorporation of AI and NLP
-
innovative research that demonstrates how AI can advance the measurement of skills such as clinical reasoning and communications
-
the potential AI holds for transforming assessment beyond multiple-choice questions to provide personalized feedback and greater facilitation of learning
-
NBME’s approach to and development of ethical guidelines
The presentation was followed by a live Q&A session, during which attendees submitted questions for the presenters.
Watch the full recording to learn more.
Highlights from the presentation
Read a summary of the presentation's key points.
Dr. Andrea Anderson, MD — Moderator
Associate Chief and Associate Professor of the Div. of Family Medicine, The George Washington University School of Medicine and Health Sciences
The Chair of the DC Board of Medicine, Dr. Anderson has been active in local and national health policy and medical regulation as well as teaching ethics, professionalism, and physician advocacy to medical students and residents. She serves on the national Ethics and Professionalism Committee of the Federation of State Medical Boards (FSMB), the State Medical Board Advisory Panel to the USMLE and other USMLE national committees of NBME.
Victoria Yaneva, PhD — Presenter
Manager, NLP Research
Victoria has a PhD in NLP and is interested in exploring the intersections between NLP and educational measurement for the purpose of developing innovative solutions for high-stakes clinical exams. Some of her research focuses on:
-
Automated scoring of physician notes and short-answer questions
-
Automated test item generation and distractor suggestions
-
Predicting test item characteristics and quality from text
She has published over 40 papers on NLP topics such as educational applications, readability, accessibility and modeling of behavioral data. She is an editor of the book Advancing Natural Language Processing in Educational Assessment.
Kimberly Swygert, PhD — Presenter
Director, Test Development Innovations
A psychometrician for 25 years, Kimberly has a PhD in Quantitative Psychology and currently leads the NBME Test Development team that supports innovative technologies and projects. Her research and operational interests include:
-
Development of multiple-choice questions and other innovative item types
-
Automated item development via the use of cognitive modeling
-
The use of innovative technologies such as NLP to support improved test development and test security processes
Kimberly regularly presents at conferences and has published chapters on item and test development in recent editions of Assessment for Health Professions Education, the Handbook of Test Development and the Guidelines for Technology-Based Assessment.