News & Events
Nearly 300 medical school faculty to attend 2nd Annual NBME Invitational Conference for Educators
Medical school faculty are critical to the education of future physicians. In addition to teaching, faculty must assess their students’ knowledge to know if students are meeting objectives set for courses and clerkships and if changes are needed to the educational program. The intensity of graduate school and medical school that most medical school faculty experienced leaves little opportunity to learn how to create good assessments. A goal of the National Board of Medical Examiners® (NBME®) is to equip medical educators with all the tools and best practices for writing better school-based assessments for their learners.
For the second year in a row, nearly 300 US medical school educators will participate in NBME’s Invitational Conference for Educators being held this year in Indianapolis on May 15-16. The event will enable faculty to network with each other and learn how to write better assessments through two assessment tracks - Written Assessment Track and Performance Assessment. Both tracks will be led by expert NBME staff to furnish participants with the knowledge and current techniques for building new skills.
Last year’s conference key-note speaker, Alison Whelan, Chief Medical Officer at the Association of American Medical Colleges (AAMC), said “When assessment is done right, it is helping students to see their progress and become independent physicians.”
She also noted “Why do we assess in medical education? It’s for our learners. It’s for our programs- are we meeting our goals? It is for others. We assess so our students can get licensed. And most importantly, we do it for the public. We have to show that the people we are training to care for them have the competence and the abilities to do what they want to do. “
At the conclusion of last year’s event, many participants expressed the desire to volunteer with NBME in the future and felt “motivated and excited to share with my colleagues and to put what was learned into action.” This year’s conference aims to make the same impact by highlighting the value of assessment and deepening the knowledge and skills of the community.
NBME is committed to providing resources and benefits to the medical education community in recognition of the valuable contributions of the faculty who give their time and energy to our products and services. More than 400 individuals serve on a United States Medical Licensing Examination® (USMLE®) committee, panel, or task force each year. While NBME has had item-writing workshops and educator guides available for several years, leaders and staff wanted to do more. In 2017 NBME’s Executive Board endowed the Strategic Educator Enhancement Fund to sustain annual conferences and other activities to help NBME engage with, and support, the medical education community. We are grateful for the extraordinary commitment and dedication of faculty members in advancing NBME’s mission.
To see more of the resources that NBME has for medical educators, click here.
New Fellowship Announced for Medical Education Research
The National Board of Medical Examiners® (NBME®) is proud to announce a unique research opportunity for medical school faculty who wish to develop their skills in medical education assessment research.
The Strategic Educator Enhancement Fund (SEEF) Medical Education Research Fellowship Selection Committee, sponsored by NBME, is now accepting letters of intent for a specially designed cohort of eight participants. The cohort will bring together like-minded participants with mentors and subject matter experts to develop a research protocol and research program for one or both of the following topics:
- assessment of clinical reasoning/critical thinking and/or
- diversity/bias and disparities in assessment.
The new SEEF Medical Education Research Fellowship holds many benefits for the medical education community including the creation of new assessments that can be used in the continuum of physician learning. The goal of the Fellowship is to provide an opportunity to US medical school faculty who:
- Are interested in developing skills in medical education assessment research
- Have a commitment to working with a team of interested colleagues
- Are interested in assessment-focused scholarship about innovative approaches that will enhance the assessment of individuals across the continuum of medical education and in active independent practices
To learn more about the Fellowship, view our online program information here.
Psychometric Staff Win Data Competition
Two National Board of Medical Examiners® (NBME®) psychometricians have received recognition for their work in data visualization. Richard Feinberg, PhD, and Dan Jurich, PhD, recently won the Educational Measurement Issues and Practices (EM:IP) cover graphic/data visualization competition for the fourth consecutive year with their entry Visualizing Hierarchical Score Inferences.
Psychometricians help ensure validity throughout the process of designing, building, scoring and reporting performance on assessments by being mindful of the inferences that can be made by users of the score information. Their winning graphic illustrates different inferences that may be desired by examinees and how reporting performance within these different contexts can change the interpretations made from the score report. This work reinforces the NBME’s commitment to understanding what’s meaningful to examinees, and using that understanding to create useful score reports that facilitate evidence-based decision making.
EM:IP is one of two prominent journals run by the National Council on Measurement in Education and a leading forum for measurement and psychometric research.
NBME Executive Board Election
The NBME is pleased to announce newly elected and re-elected officers and at-large members of its Executive Board. The election occurred at the annual meeting of the NBME membership on March 29, 2019, in Philadelphia.
The following individuals were elected/re-elected to Executive Board:
- Alfred F. Tallia, MD, MPH: Elected to a first two-year term as chair
- Paul M. Wallach, MD: Elected to a second two-year term as vice chair
- Latha Chandran, MD, MPH: Elected to a first two-year term as treasurer
- Donna D. Elliott, MD, EDD: Elected to a second two-year term as an at-large member
- Marie C. Foley, PHD, RN, CNL: Elected to a first two-year term as an at-large member
- Reena Karani, MD, MHPE: Elected to a second two-year term as an at-large member
- Pat Mastors: Elected to a second two-year term as an at-large member
- Karen M. Sanders, MD: Elected to a first two-year term as an at-large member
- David Milling, MD: Elected to a first two-year term as an at-large member
- Kamili Wilson: Elected to a first two-year term as an at-large member
Suzanne T. Anderson serves as immediate past chair, and NBME president and CEO, Peter J. Katsufrakis, MD, MBA, is an ex officio member of the Executive Board.
Profiles of NBME’s Executive Board members are available here.
2018 NBME Annual Report Available
The 2018 NBME Annual Report: Engagement and Action and 2018 Directory: NBME & USMLE Committees & Volunteers are now available for viewing. Click here to see the report’s interactive format that quickly transports readers to deeper content or directly to online assessment information.
This year’s report spotlights progress achieved through NBME’s engagements and assessment improvements. You’ll also read about updates to programs and services, recognition and grants, and assessment-related research.
Lively infographics offer fast facts about a variety of topics. For more information regarding the topics within, please contact our Director of Communications at firstname.lastname@example.org.
Engaging with Faculty at Item-Writing Workshops
On March 4, 2019, NBME’s Director of Test Development Services (TDS) conducted a workshop for 20 participants at the American College of Obstetricians and Gynecologists (ACOG). For the past 10 years ACOG has reached out to NBME for a spring workshop to train subject matter experts writing items for ACOG’s PROLOG (self-directed learning) and CREOG (residency in-training) assessments.
NBME has developed test items and examinations for licensure and other assessment purposes since 1915. NBME staff members are internationally recognized experts in the field of test development, psychometrics, and medical education research. For over 20 years, members of NBME’s professional test development staff have served as a resource to medical school faculty wishing to improve their item-writing techniques. Additionally, faculty from other health professions (e.g. veterinary, physical therapy, podiatry) reach out to NBME to conduct workshops to improve their assessments.
As part of its goal of collaborating with organizations to develop better assessment tools for learning, measurement, education, and patient outcomes, TDS conducts many off- and on-site domestic workshops throughout the year. In 2018, TDS facilitated 18 offsite workshops, in addition to many onsite workshops for the United States Medical Licensing Examination® (USMLE®) and its client exam programs. For 2019, 16 have already occurred or are planned through mid-year.
The item-writing workshops are designed to help faculty members improve the quality of multiple-choice questions written for their examinations. Through informative presentations and discussions, faculty members are able to explore structural issues relating to constructing quality multiple-choice items for testing of basic science and/or clinical science content. An overview of item formats concentrating on the traditional one-best-answer item formats that require examinees to select the single best response is presented. One-best-answer questions are the most widely used multiple-choice-item format. Several examples of well-constructed one-best-answer items are presented. Each consists of a focused item-stem (e.g., a clinical vignette) and a lead-in question, followed by a series of options with one correct answer for each item. Several examples of technical item flaws are presented and discussed during the session.
Each participant is provided with a printed copy of the manual “Constructing Written Test Questions for the Basic and Clinical Sciences,” which is published by NBME. The manual includes the material that is presented during the session and additional templates that can be used to assist item writers develop items. Copies of this manual can be obtained at www.nbme.org.
The workshop day consists of a short educational session followed by a rewriting exercise. NBME provides workshop participants with a handout of poorly structured items (constructed by our own in-house medical experts); participants rewrite items in small groups, and then present the rewritten items to the full group. The group discusses and provides both content and structural suggestions. The facilitator will also display the NBME version of the rewritten item.
Feedback from the workshops has been very positive. Participants have expressed that they feel that these workshops improve their ability to create their own assessments, recognize test question flaws, and rehabilitate poor questions. Participants are surveyed at the end of the session to assess their knowledge gained and obtain feedback on how sessions can be improved.
You can learn more about how you can have NBME conduct a workshop at your institution here.
NBME Welcomes New Vice President of Test Development
The National Board of Medical Examiners® (NBME®) is pleased to announce that Ken Ridgley joined the organization as Vice President, Test Development in mid-February 2019.
“Ken’s industry knowledge of innovative approaches, processes, and systems is the right complement for the momentum building in NBME’s continuous improvement journey,” said Craig Mills, Senior Vice President. “His desire to support our mission of serving the public health through state-of-the-art assessment makes his selection all the more appropriate.”
Through Ken’s work with a diverse assessment client base, he has gained experience in alternative methods of item development across multiple industries, which will benefit all NBME examinations. His expertise with industry-leading technology solutions for test-item databases, authoring, and exam construction all align with NBME’s objectives to increase value in its current examinations, explore alternative assessments, and discuss new approaches and competencies to serve the healthcare consumers.
Mr. Ridgley was previously Vice President, Test Development for Prometric where he worked for ten years leading their Global Test Center Support function, Operations Command Center and most recently, Test Development. Prior to this, he spent 10 years in progressive leadership roles managing large scale, global teams in process improvement, change management, strategic planning, and organizational development.
Coming Soon: New Comprehensive Basic Science Self-Assessment Forms 20, 21, and 22
Three new practice exam forms (Forms 20, 21, and 22) of the Comprehensive Basic Science Self-Assessment (CBSSA) will be available on MyNBME on March 25, 2019. Two additional new forms (Forms 23 and 24) will be released in late spring 2019. The performance feedback provided will remain the same. These updates are part of NBME’s annual operations to provide new content.
CBSSA Forms 13, 15, 16, 17, and 19 will be retired when the new forms are released. CBSSA Form 18 will continue to be available for purchase. Participants who purchased CBSSA Forms 13, 15, 16, 17, or 19 prior to March 25 will still have 90 days from date of purchase to launch and complete their self-assessment. For example, if you purchased Form 19 on March 24, the self-assessment will be available to you until June 21, 2019.
NBME Educator Guides Now Available on Test Blueprinting and OSCE Topic Areas
Health profession educators can now learn about assessment creation and delivery through additional NBME Educator Guides now available at NBME.org.
Previously, these lessons were available for purchase, but now are available for free as "companion" lessons to complement the NBME "Constructing Written Test Questions for the Basic and Clinical Sciences" (sometimes referred to as the Gold Book).
The seven companion lessons cover topics not included in the NBME Gold Book and include content on test blueprinting as well as several clinical skills topic areas.
The NBME believes that medical school faculty should have access to high-quality professional development resources that address quality practices in assessment. To accurately assess student understanding, faculty training resources are very important, especially as it has become common practice for medical students to learn required content from many sources, including outside the classroom.
Commitment to quality assessment is fundamental to us, and we are excited to offer these additional lessons for free. In addition to the Gold Book, these lessons will provide additional education and guidance in multiple areas of assessment. We are pleased to offer these lessons for use in medical education and assessment.
Lessons are available here.
New Website for NBME Self-Assessment Services Benefits Examinees
A new customer website, MyNBME, is now available for examinees who are taking NBME Self-Assessment Services (NSAS) to gauge their readiness for the United States Medical Licensing Examination® (USMLE®), NBME Clinical Science Subject Exams, or the IFOM Clinical Science Examination.
MyNBME was designed based on feedback obtained from students and residents during focus groups and pilot trials. To gather the information, customer journey maps were completed, and experience surveys were deployed to benchmark improvement efforts. Both students and residents told us they wanted to quickly and easily:
- purchase products across different product lines in a single transaction
- view and download score reports
- complete other activities and communications, not only with NBME but also with collaborators
Examinees are encouraged to share their feedback so NBME can keep getting better. If you have any questions regarding NSAS, please contact us here.
CDC grant awarded to NBME Collaborator
The International Consortium for Health & Wellness Coaching (ICHWC), a nonprofit controlled affiliate of the National Board of Medical Examiners (NBME), recently received a grant from the Centers for Disease Control and Prevention (CDC) for a coaching program designed to help prevent diabetes. The organization will partner with the American College of Preventive Medicine (ACPM) to develop the following:
- group coaching competencies
- a training program
- a certificate examination
This work will support dissemination of the National Diabetes Prevention Program lifestyle change program. NBME staff views the collaboration on these high-quality certification examinations as having great impact on the organization’s mission. Learn more here.
Philadelphia READS and NBME reads with it
NBME has always looked for ways to be involved with the local community. In 2008, a staff member suggested the company participate in the Philadelphia READS' "Power Partners" program, which matches socially conscious organizations with children from Philadelphia classrooms for one-on-one literacy coaching sessions that run from October through May of each school year. During weekly visits to NBME, these children receive both academic support and exposure to positive adult role models.
Since that time, NBME volunteers have worked with a number of different schools. Volunteers become reading coaches for the program and are given the opportunity to spend one-on-one time with a child. In addition to reading books, the coaches get to know the child they are partnered with, sharing information about the world, their own lives, and common topics of interest. This one-on-one interaction helps the students to gain confidence as their reading level increases.
"When you have a breakthrough with your reading partner, it is really rewarding," says Jesse Delaney, a senior case developer for NBME. "Sometimes that breakthrough is as simple as finding a topic the student enjoys and tailoring the reading list to that interest. Other times, it is seeing a student flawlessly and effortlessly read a word aloud that he or she struggled with earlier in the year."
Pat Callahan, program support specialist for Philadelphia READS, says that the program allows the children to interact with positive role models and experience authentic volunteerism in the workplace. The students value their time with their reading partners, with teachers noting that attendance seems to be highest on the days when they are scheduled to participate in the program. NBME staff members are equally excited on those days and take great joy in making a difference in a young person's life.
Philadelphia READS is a child literacy nonprofit program with a mission of providing access to books and community programs to foster a love of reading and increase literacy in Philadelphia. NBME is proud to be a part of a program that generates and encourages the love of reading in Philadelphia.
NBME Brings Health Profession Organizations Together for Collaborative Approaches in Continuing Certification
NBME hosted an invitational conference on Collaborative Approaches in Continuing Certification in December for key collaborators.
"The meeting afforded attendees opportunities for connecting with colleagues, deepening the conversation about continuing certification, and affirming the importance of initial and continuing certification in their professions and for the public they serve," said Lisa Rawding, Senior Vice President of Assessment Programs.
Pat Mastors, founder of Patients' View Partners and NBME Executive Board member, opened the event with a rousing presentation to the group. She challenged attendees to consider how to help patients more readily understand the competence of the medical providers that they are choosing. Using examples from around the globe, Pat highlighted the opportunity for healthcare professionals to take ownership of communicating the value of certification.
The main program included presentations and small group activities designed to explore goals, challenges, and opportunities in continuing certification. Representatives from allied health, physician, and veterinary communities worked together to explore key themes raised during the focus group discussions. For many it was the first opportunity to explore these questions with colleagues from different specialties. "We heard resoundingly positive feedback from participants about how valuable it was to hear the experiences of colleagues across varied professions," said John Phebus, Assessment Officer.
NBME greatly appreciated the opportunity to provide a forum for this important discussion in support of the health professions we serve. To learn more about NBME's services, please contact Amber Williams at email@example.com.
NBME Collaborates on a New Evidence-Based Interprofessional Professionalism Assessment Tool
The Interprofessional Professionalism Collaborative (IPC) is pleased to announce the publication of its landmark article – The Intersection of Professionalism and Interprofessional Care: Development and Initial Testing of the Interprofessional Professionalism Assessment (IPA). The final published article is now on Taylor and Francis Online and will be assigned to a future issue of the Journal of Interprofessional Care. The IPA instrument was created by the IPC, a national group representing 11 entry-level health professions and the NBME.
"NBME is honored to have collaborated on the Interprofessional Professionalism Assessment and its accompanying tool kit. This work advances assessment of interprofessional professionalism and collaboration among health professionals," said Dr. Peter Katsufrakis, President and CEO of the National Board of Medical Examiners. "We serve the public through state-of-the-art assessment of health professionals, and we remain committed to research and development in evaluation and measurement."
Interprofessional Professionalism is the consistent demonstration of core values evidenced by professionals working together, aspiring to and wisely applying principles of altruism and caring, excellence, ethics, respect, communication, and accountability to achieve optimal health and wellness in individuals and communities.
"The ability of health care practitioners to demonstrate interprofessional professionalism is essential to the delivery of quality care to patients, families, communities, and society," said Dr. Jody Frost, Education Consultant and IPC Facilitator. "As profiled in the Journal of Interprofessional Care, the new psychometrically sound assessment instrument developed by the IPC can be applied in both academic and practice settings to cultivate environments that promote interprofessional professionalism and collaborative care models."
The IPA measures observable behaviors of health care professionals-in-training that demonstrate professionalism and collaboration, which are essential to person-centered care. The instrument was created and evaluated over several years through a comprehensive, multi-phasic process, which included the development of a construct and observable behaviors; instrument design, expert review, and cognitive interviews; and psychometric testing.
"As a pharmacist, who worked on multidisciplinary health care teams, I believe the construct of interprofessional professionalism is integral to successful collaboration among health care professionals in providing optimal patient care," said Jennifer Wise, NBME representative on the IPC. "I am proud to be a member of the IPC and believe the IPA and the accompanying Tool Kit will be great educational resources."
The IPA contains 26 items or observable behaviors representing six domains of professionalism (altruism and caring, excellence, ethics, respect, communication, accountability), and was tested by 233 preceptors who rated health profession learners in the final year of their practical training.
In addition to the new journal article, the IPC has created an online tool kit to facilitate interprofessional professionalism, which includes the IPA Instrument, case scenario videos, an on-demand webinar, and related materials. To access these resources, visit http://www.interprofessionalprofessionalism.org.
RENEW Task Force Explores Effects of Stress on Examinees at Recent Session
The Re-Examining Exams: NBME Effort on Wellness (RENEW) Task Force held a successful meeting at the end of October. The task force made progress in fulfilling its charge. To learn more, click here..
M. Brownell Anderson honored with AAMC GEA award in her name
The Group on Educational Affairs of the Association of American Medical Colleges announced a new Award for Emerging Educators. It is named in honor of NBME's M. Brownell (Brownie) Anderson, M.Ed., for her outstanding commitment to the Group on Educational Affairs (GEA).
The purpose of the GEA is to advance medical education and medical educators through faculty development, curriculum development, educational research, and assessment in undergraduate, graduate, and continuing medical education. Brownie began her work with them in 1983 and continued that work for more than 25 years. In that time, in collaboration with many medical educators in all of the schools, she increased the attention to and support for medical education at the AAMC.
The AAMC has a number of affinity groups for faculty and staff of AAMC member medical schools, teaching hospitals; the Group on Educational Affair is one of those groups. NBME President and CEO, Peter Katsufrakis, was the national chair of another of AAMC's affinity groups: the Group on Student Affairs (GSA).
Brownie began working at the National Board of Medical Examiners (NBME) in 2011 and the GEA wants to show recognition for her long-standing work with and commitment to the community of medical educators. They also want to honor her uncanny ability to identify, encourage, and welcome individuals early in their careers to the community of medical education.
The GEA states: In the spirit of M. Brownell Anderson, this award recognizes early career medical educators who have demonstrated enthusiasm, passion, commitment and creativity in the field of medical education. This award identifies and supports emerging educators in their career trajectory in medical education and serves as an opportunity to recognize and validate their potential and talent in the field. The GEA owes a debt of gratitude to M. Brownell Anderson for recognizing and cultivating new leaders for many years, establishing a tradition that lives on through this award.
In Brownie's own words, she expressed "I am overwhelmed, humbled, and a bit terrified by having this award named for me and above all, incredibly grateful. All I did was ask people who inspired me to help me get the work done. I am so fortunate to have been given the opportunities I was given and to have been able to work with and learn from so many incredible people."
Official nomination information and criteria will be made available in early 2019.
NBME Way in Action: Improving Primum
When you have a continuous improvement mindset, a question that occurs often is, "How could this be better?" That's exactly the question Nicole Fortuna, Senior Test Development Analyst, asked herself regarding preparation materials for the Computer-based Case Simulation (CCS) portion of the USMLE Step 3 examination. The result of her question was a new video, which you can view here.
Based on end-of-survey comments, staff found that not all examinees understand how to use the Primum® software on which the Step 3 examination runs. They reported dissatisfaction with the practice materials on the website; for example, while downloadable to PC users, the practice cases were not downloadable to Mac users. Staff began to understand that some examinees were experiencing a disconnect with the software because of a lack of quality educational materials.
After applying continuous improvement principles, Nicole arrived at an idea: to create a video that would support a better understanding of the Primum software.
"When I mapped out possible solutions, the best solution wasn't to get rid of any materials," she said. "It was to provide a more seamless piece of technology that anyone can use anywhere in order to prepare to better navigate the software on exam day."
She hopes that by creating better ways to prepare students, and by sharing this through our social media channels and website, we'll be better able to serve our examinees and to ensure that we are their first stop for information.
NBME CEO participates in AMA initiative to transform residency training
In late October, the American Medical Association (AMA) announced a new "Reimagining Residency" initiative and its committee members who will work to transform residency training and better address the workforce needs of the current and future health care system. The five-year, $15-million grant program will support innovations that provide meaningful and safe transitions from undergraduate medical education (UME) to graduate medical education (GME). The program will establish new curricular content and experiences to enhance readiness for practice and promote well-being along the continuum of medical training.
NBME is in full support of this initiative with Dr. Peter Katsufrakis, President and CEO, stating:
“The AMA’s leadership over the past five years with its Advancing Change in Medical Education initiative demonstrated the power of convening a collaborative of educators committed to enhancing education. This experience naturally leads to considering how best to help transform residency training. The NBME’s critical role in ensuring the qualifications of health care providers aligns fully with the AMA’s support to enhance residency training. I am happy to help guide the development and implementation of this initiative.”
The NBME is an independent, mission-driven organization that provides high-quality assessments for the health professions. We serve the health of the public through state of the art assessment of health professionals, and we are committed to research and development in evaluation and measurement.
NBME Psychometrician Joins ATP Board of Directors
The NBME is pleased to announce that Kimberly Swygert, PhD, Director of Research and Development in Professional Services, has been elected to the 2019-2020 Association of Test Publishers (ATP) Board of Directors.
The NBME is a long-standing member of ATP. According to its website, ATP "is a non-profit organization representing providers of tests and assessment tools and/or services related to assessment, selection, screening, certification, licensing, educational, or clinical uses. ATP members are pledged to promote and advance the integrity of assessment services and products and their value to society; and dedicated to the highest level of professionalism and business ethics within the test publishing community."
"I am delighted and honored to be elected to serve in this role and I'm greatly looking forward to representing the Certification/Licensure field in upcoming discussions of topics, such as new industry standards, recertification, test security, best practices and other important areas of exploration and research." she said.
Dr. Swygert is the co-editor and co-author of the most recent edition (2017) of best practices in item writing for medical educators, Constructing Written Test Questions for the Basic and Clinical Sciences. In addition to her work at NBME, Dr. Swygert has taught graduate courses in biostatistics and psychometrics for Drexel University and the Uniformed Services University of the Health Sciences, where she is currently on an external advisory board.
Andrew F. Farella to Join NBME as Sr. Vice President of Information Technology
Posted: October 18, 2018
Following a nationwide search, the National Board of Medical Examiners® (NBME®) is pleased to announce Andrew F. Farella will become Sr. Vice Present of Information Technology in November 2018.
"Andy's experience and perspective were just what we were looking for, an ideal balance that will enable him to fit in well at the NBME while also bringing new ideas and a fresh point of view to our organization," said Peter J. Katsufrakis, MD, President and CEO of NBME.
Mr. Farella is currently Associate CIO, AVP Business Applications of Children's Hospital of Philadelphia where he spent the last nine years leading teams focused on implementation and support of the patient access, CHOP's web presence (SharePoint, intranet, internet site), custom application/mobile development, financial systems, and enterprise business applications. His experience also includes almost a decade serving as Director, IT Applications North America for Elan Pharmaceuticals.
His scope of strategic leadership will include Application Services, Infrastructure Services, IT Project Management, IT Security, Disaster Recovery and Compliance, IT Continuous Improvement and Contract Management.
More Good News for Medical Students about NBME Self-Assessments
If you are preparing for the NBME Clinical Science Subject Examinations or the IFOM Clinical Science Examination, learning from your mistakes and reviewing those questions you guessed at during practice tests can be just as important as reviewing content. Together, these strategies offer you a great opportunity to fully understand the broad array of concepts required for your test.Read the full article
NBME Psychometrician Recognized
Congratulations to Kimberly Swygert, PhD, Director of Research and Development, for being selected as a Fellow of the College of Physicians of Philadelphia. The Admissions Committee and the Board of Trustees recently announced their decisions, and Dr. Swygert will be inducted via the College Night ceremony at the Mütter Museum on Friday, November 16. Fellows of the College of Physicians of Philadelphia are selected for their scientific contributions to work that advances the understanding of medicine. Kimberly is thrilled to join this distinguished group that already includes two physicians at the NBME – our president, Dr. Peter Katsufrakis; and our Medical Advisor, Dr. Miguel Paniagua.
Dr. Swygert’s work on performance assessments, examinee timing and pacing, examinee repeater behavior, and score reporting has been presented at conferences and published in journals such as Academic Medicine, Advances in Health Sciences Education, the Journal of General Internal Medicine, and the Journal of Educational Measurement. Her chapter on performance assessments in the content of credentialing examinations appeared in 2015 in the updated Handbook of Test Development. She is the co-editor and co-author of the most recent edition (2017) of best practices in item writing for medical educators, Constructing Written Test Questions for the Basic and Clinical Sciences. In addition to her work at NBME, Dr. Swygert has taught graduate courses in biostatistics and psychometrics for Drexel University and the Uniformed Services University of the Health Sciences, where she is currently on an external advisory board.
New Question Formats Available
The Medicine Clinical Science Subject Examination now contains several short-answer questions to closely reflect day-to-day physician decision making. The new items allow examinees to provide short, free-text answers to assessment questions.
Medical schools use NBME subject examinations in the basic and clinical sciences to evaluate the educational achievement of individuals in specific subject areas. The examination results may provide insight into the educational progress of physicians or other health professionals.
The NBME is committed to developing innovative item formats that more closely reflect clinical practice. As part of this commitment, the exam now features up to five short answer questions in a given administration. When making a clinical decision, physicians use their knowledge and training to make decisions based on the clinical information available. Short-answer items are designed to more closely reflect this decision making process.
The Medicine Clinical Science Subject Examinations are administered at both medical schools and Prometric centers. Instructions for practicing with these new item formats are provided in the exam tutorial.
Tutorial instructions and a sample item are available here.
For your reference and convenience, we have provided frequently asked questions.
If you have any questions, contact us at firstname.lastname@example.org
New NBME Presence at the AMEE Conference
NBME staff members are excitedly preparing for the Association for Medical Education in Europe (AMEE) conference later this month. Here’s how we’re showing up in a new way:
- New booth. In addition to making our booth more inviting, we’ve created a question card that features questions from the International Foundations of Medicine (IFOM) self-assessment. Can You Answer A Question correctly?
- New opt-in card. Now that the European Union’s General Data Protection Regulation (GDPR) is in effect, we created an opt-in card so you can receive updates from us.
- New brochure. In addition to information about our international products (IFOM and other customized international examinations), our new brochure includes information about other helpful offerings, including Subject Exams, Customized Assessment Services, NBME Self-Assessment Services, and Item Writing Workshops.
Will you be there? If so, we can’t wait to meet you.
Correct Answers Available for NBME Self-Assessments Beginning July 2
No matter how much you prepare, sometimes the correct answer may not be apparent to you. And later you may not remember the question or the answer you chose.
If you are preparing for a United States Medical Licensing Examination® (USMLE®) Step, learning from your mistakes, or going over those questions you guessed at during practice tests can be just as important as reviewing content. Together, it offers you a great opportunity to fully understand the broad array of concepts required.Read the full article
The Stemmler Medical Education Research Fund issues its 2018-2019 call for letters
In support of innovation in medical education assessment, the Edward J. Stemmler, MD Medical Education Research Fund from the NBME has issued its 2018-2019 call for Letters of Intent.
The purpose of the Stemmler Fund is to provide support for research and development in innovative evaluation methodologies or techniques, with the potential to advance assessment in medical education or practice.
Applicants should consider how their proposed research provides a departure from previous thought and practice, and advances the field of assessment in medical education. The application deadline is July 15, 2018 Applicants may visit here for more detailed information.
Seminal Research by NBME Staff Announced
NBME staff publications and presentations contribute to the body of scholarship on assessment, explore emerging test constructs, and demonstrate the continuous reliability and validity of our existing examinations.
A comprehensive list of the work created by staff can now be found under Research on NBME.org. From the 1900s through the 2010s, staff members have made great contributions to the seminal research in assessment. The work is presented by decade, with an introduction that delineates the themes that emerged from that time period. Both the publications and presentations made at national and international conferences and meetings are included.
For those contributing to the body of knowledge about assessment, this list provides a useful starting point before doing a deep dive into the literature.
Measurement Experts at NBME Receive National Recognition
Three National Board of Medical Examiners® (NBME®) researchers have received recognition for their influential studies in psychometrics, the science of measurement. Drs. Brian Clauser, Richard Feinberg, and Carol Morrison have been honored with awards for career and research contributions.
Read the full article
New Score Scale for the Emergency Medicine Advanced Clinical Examination
Register Below to Learn More
Beginning with exam administrations on June 1, 2018, the National Board of Medical Examiners® (NBME®) will report scores for the Emergency Medicine Advanced Clinical Examination as equated percent correct scores instead of scaled scores. The new equated percent correct score reflects an examinee's mastery of the content domain and can be used to compare and track school and examinee performances over time and across test administrations.
This new scale will also help to enhance the quality of score reporting and interpretation for the Emergency Medicine exam.
Informational webinars for medical school faculty who want to learn more about the changes are being made available by NBME Psychometricians Linette Ross and Seohong Pak.
Register here to join a webinar. (Sessions will close when they reach capacity.)
You can also send questions to us at email@example.com.
2019 John P. Hubbard Award Call for Nominations
The NBME invites nominations for the 2019 John P. Hubbard Award. This award recognizes individuals who have made a significant and sustained contribution to the assessment of professional competency and educational program development at any level along the continuum of medical education and delivery of healthcare.
The Hubbard Award recognizes significant and sustained contributions to the pursuit of excellence in the field of evaluation in medicine. Recognition of contributions to medicine in curricular development, instructional methodology, or medical training methods is not the primary purpose of the award.
Read the full article
Health and Wellness Coach Certifying Exam Helps Demonstrate Excellence
This June, hundreds of potential health and wellness coaches will take the Health and Wellness Coach Certifying Examination (HWCCE) to demonstrate their competency in helping people achieve their personal goals for health and wellbeing.
National Board of Medical Examiners® (NBME®) leadership and executive leaders from the International Consortium for Health & Wellness Coaches (ICHWC) collaborated on the first national certification for individual health and wellness coaches.
Read the full article
NBME Collaborates with Hospital Sírio-Libanês
After a recent meeting in Brazil to discuss its ongoing collaboration activities with NBME, Hospital Sírio-Libanês (HSL) hosted a symposium with attendees from the student body and faculty of a number of medical schools, government officials, and other healthcare luminaries. The symposium, presented in Portuguese with a simultaneous English translator, provided insight into the NBME/HSL collaboration, the Qualifications in Medicine (QM) exams, and progress toward fostering a culture of assessment in Brazil.
“In the US we have a culture of assessment,” said M. Brownell Anderson, Vice President for International Programs . “We use examinations to make decisions about people’s progress in educational settings, their qualifications to work in a particular field, drive a car, or offer a service.”
Brazil is seeking to build a culture in which healthcare professions can demonstrate the appropriate standard of care.
Read the full article
Employees donate art supplies to Children's Hospital through Lucy Belle's Rainbow
From March 12 through April 16, employees at the National Board of Medical Examiners® (NBME®) collected art supplies for "swag bags" donated to children undergoing treatment at a local children's hospital.
The NBME proudly supports employee involvement in the community through a dedicated "Volunteering at NBME" group, which fundraises for and builds relationships with several local organizations. The NBME also provides employees with paid volunteer hours that employees are encouraged to use to support causes about which they are passionate.
The NBME worked with Lucy Belle's Rainbow for the collection of crayons, sketchbooks, watercolor kits, glue sticks, markers, colored pencils, and other supplies. Lucy Belle's Rainbow is an initiative to bring the power of art and art therapy to children undergoing cancer treatment, in an effort to help children and their families cope with stress. The organization was inspired by the memory of Lucy Belle Perkins "whose love of art and all things beautiful will always exist."
To learn more about Lucy's Belle's Rainbow, including where you can donate or how to become involved, click here.
Education grants awarded to organizations in Brazil and Mexico
The National Board of Medical Examiners® (NBME®) Latin America Grants Program awarded grants to three collaborations of Latin American medical schools for their proposals to enhance the quality of evaluation in health professions education in their respective regions. The Latin America Grants Committee selected the recipients in February after receiving more than a dozen proposals.
Over its 103-year history, the NBME has pursued and maintained the highest standards of excellence in its examination programs. As part of its Centennial celebration in 2015, the NBME supported three consortia of Latin American medical schools in their work to develop evaluation capacity in their countries. Building on the success of that program, the Latin America Grants Program presented the three recent awards.
Each proposal selected includes ideas for sustainable faculty development activities, creation of a Massive Open Online Course (MOOC), and development of a methodology for longitudinal assessment of abilities and skills acquired over the course of a medical internship. The proposals are designed to have a broad, sustainable impact by building local capacity to evaluate key learning outcomes in the education of those pursuing health professions. They are also expected to improve program quality at participating schools.
Read the full article
RENEW Task Force Seeks to Support Examinee Wellness
The National Board of Medical Examiners® (NBME®) is committed, along with like-minded organizations, to identifying potential sources of burnout for physicians in training and contributing to solutions. In February, the NBME convened the inaugural Re-examining Exams: NBME Effort on Wellness (RENEW) task force. The organization’s goal in this initiative is to address the challenge of physician wellness and to acknowledge the stress caused by working in the health professions that begins during the educational and training processes.
The RENEW Task Force was created to:
- Understand the role high-stakes exams, such as the United States Medical Licensing Examination® (USMLE®), as well as secondary uses of those exams, play in affecting medical student and physician wellness.
- Develop and execute a research agenda that will lead to actionable interventions to promote wellness and mitigate burnout.
- Support actionable interventions to promote wellness amongst learners and trainees in the context of assessment.
Outreach to Students Continues
April 10, 2018 — In an effort to broaden its outreach, colleagues from the United States Medical Licensing Examination® (USMLE®) program hosted the inaugural meeting of the Medical Student and Resident Advisory Panel to the USMLE Program. The meeting, facilitated by staff from the National Board of Medical Examiners and the Federation of State Medical Boards (FSMB), was held at the FSMB’s headquarters in Euless, TX. The meeting was designed to:
- Collect feedback from medical students and residents on issues and topics specific to USMLE
- Test assumptions about the USMLE with an examinee audience
- Gain insight into the perspective of examinees on USMLE policy issues
Dr. Paul Cunningham awarded 2018 Jacobi Medallion
Executive Board member, Dr. Paul Cunningham, received the 2018 Jacobi Medallion for his contributions to the medical profession and for his achievements in academic leadership.
In addition to serving on the NBME Executive Board, Dr. Cunningham is Dean Emeritus of the Brody School of Medicine at East Carolina University. He served as Dean at Brody School of Medicine from 2008-2016.
Dr. Cunningham was awarded the 2018 Jacobi Medallion on March 15 from the Icahn School of Medicine at Mount Sinai in New York, which is also where Dr. Cunningham completed his residency in surgery.
Read the full article
NBME Announces Stemmler Grant Awards
The National Board of Medical Examiners® (NBME®) announced that the 2017-2018 Stemmler Grant award winners are from University of Michigan, University of Calgary and Uniformed Services University of the Health Sciences.
This year, three projects were chosen to receive grant funds, totaling $393,549 in funds designed to provide support for research or development of innovative assessment approaches that will enhance the evaluation of those preparing to, or continuing to, practice medicine.
Read the full article
2017 NBME Annual Report Available
Updated Educator Guide Online
Faculty can learn how to write high-quality, multiple-choice questions related to the foundational and clinical sciences by viewing NBME's updated Educator Guide.
New Histology Basic Science Subject Exam Launches July 2018
Dr. Louis Pangaro selected by NBME as Recipient of the 2018 John P. Hubbard Award
The National Board of Medical Examiners is pleased to announce that Louis N. Pangaro, MD was selected as the recipient of the 2018 John P. Hubbard Award.
Read the full article
Dr. Theodorus Jan (Olle) ten Cate selected by NBME as Recipient of the 2017 John P. Hubbard Award
The National Board of Medical Examiners is pleased to announce that Theodorus Jan (Olle) ten Cate, PhD was selected as the recipient of the 2017 John P. Hubbard Award.
Read the full article
Dr. Peter J. Katsufrakis Appointed President of National Board of Medical Examiners (NBME).
Following a national search, the Executive Board of the National Board of Medical Examiners has selected Peter J. Katsufrakis, MD, MBA to lead the NBME as its next president.
Read the full article
NBME Activities at the 2016 AAMC Annual Meeting
The NBME will host a variety of sessions in conjunction with the 2016 AAMC Annual Meeting, November 11-15, in Seattle, Washington. Activities include interactive exhibits, focus groups, demonstrations, and workshops designed to elicit ideas and feedback on USMLE and NBME assessment services. Session details are listed below. Invitations for closed sessions will be sent in early October.
Read more information
Eric S. Holmboe Selected as Recipient of 2016 John P. Hubbard Award
The National Board of Medical Examiners is pleased to announce that Eric Holmboe, MD was selected as the recipient of the 2016 John P. Hubbard Award.
Read the full article
Step 2 CS FAQs
New FAQs regarding the Step 2 Clinical Skills (CS) exam have been posted to the USMLE website.
NBME and the VA: Joining Forces to Help Veterans and Their Families
With a generous grant from the Veterans Administration, the NBME is pleased to do its part to help ensure good healthcare for military personnel and their families. Read about it here: http://khn.org/news/its-on-the-test-new-questions-require-doctors-to-learn-about-military-medicine/
NBME and the 2015 AAMC Annual and Medical Education Meetings
The NBME will host a variety of sessions in conjunction with the 2015 AAMC Annual Meeting (November 6-10) and the AAMC Medical Education Meeting (November 10-12) in Baltimore, MD. Activities include interactive exhibits, focus groups, demonstrations, and workshops designed to elicit ideas and feedback on USMLE and NBME assessment services.
Read more information
New Customized Test Battery Platform
Based on feedback from faculty, the NBME has developed the Customized Test Battery platform, which enables administration of a series of exams using a combination of fixed-form, standardized subject tests and examinations created through the Customized Assessment Services (CAS) program. The Customized Test Battery platform can expand your assessment capabilities. For example:
- If your curriculum emphasizes fundamental science in the clerkship, you can administer a clinical science subject test along with a custom module of items covering associated basic science concepts.
- You can administer a basic science subject test and supplement the exam with items from other basic science content and/or with advanced clinical vignettes to match your assessment needs.
The "Battery Builder" is housed on the Build Exam tab of the NBME Services Portal (NSP). The tool is easy to use and includes the functionality to:
- create a test battery from a list of all NBME subject tests and any customized exams that are also available in the online ordering system
- set the test delivery sequence for the selected exams
- place one or more "off the clock" breaks between exams
- lock, edit, copy and delete test batteries
Test batteries are intended for administration to a single group of examinees and delivered in one exam session. The NBME web-based testing proctoring system has been modified to reflect test battery administrations. The examinations are scored separately, and reports are posted to the appropriate subject exam or CAS folders in NSP.
Suzanne Anderson Elected Chair of the NBME
The National Board of Medical Examiners (NBME) is pleased to announce that Suzanne T. Anderson was elected to a two-year term as Chair of the National Board of Medical Examiners.
Read the Press Release
Larry Gruppen Selected as Recipient of 2015 John P. Hubbard Award
The NBME is pleased to announce that Larry D. Gruppen, PhD was selected as the recipient of the 2015 John P. Hubbard Award.
Read the full article
Phased Elimination of Paper-based Subject Examination Program Services
Updated: May 8, 2015
The NBME transitioned all basic science and clinical science subject examinations to a web-based format in 2012. In October 2014, all schools were advised to transition to web-based testing for the upcoming 2015-2016 academic year.
Plan to Eliminate Paper-based versions of the Clinical Science Subject Examinations
The paper-based versions of the Clinical Neurology, Medicine, Obstetrics and Gynecology, Pediatrics, Psychiatry, and Surgery subject examinations are at the end of their useful lives as measures of current knowledge and practice. During the upcoming 2015-2016 academic year, NBME staff plan to actively manage the reduction and ultimate depletion of existing inventories of these outdated paper exams. It is projected that the current stock of these examinations will begin to run out after the end of the 2015 calendar year.
Beginning in January 2016, the paper-based versions of these examinations originally developed in 2010 will be retired. As this will further limit the inventory of paper examinations in rotation, there will be increased likelihood that schools may receive a recently administered form. As long as stock remains in these Clinical Science disciplines, we will continue to provide schools with these examinations in paper-based format through June 2016.
Reminder about Eliminating Paper-based Subject Examination Program Services
As a reminder, the first phase of the discontinuation of paper goes into effect August 1, 2015, when the following examinations will no longer be available for administration via paper-based format:
- Introduction to Clinical Diagnosis (added)
- Family Medicine
- Comprehensive Clinical Science
- Behavioral Sciences
- Gross Anatomy
- Histology and Cell Biology
Also effective July 23, 2015, expedited score reporting (a.k.a. "Rush Score Reports") for paper-based examinations will no longer be offered.
If you have any questions, please contact the NBME Subject Examinations team at firstname.lastname@example.org.
Characteristics of Clinical Clerkships: NBME Survey Results
The National Board of Medical Examiners (NBME) is pleased to announce that detailed results of the Characteristics of Clinical Clerkships Survey are now available. In 2012, the NBME emailed Executive Chief Proctors at over 200 medical schools and satellite campuses asking them to forward an invitation to complete an on-line questionnaire to the clerkship directors in:
- Clinical Neurology
- Family Medicine
- Obstetrics and Gynecology
The purpose of the survey was to gather descriptive information about clerkship characteristics and use of computer case simulations. Data gathered for each clerkship included type and length of clerkship rotations, curriculum type for the 2012-2013 academic year, purpose of the exam, and for the 2011-2012 academic year, procedures used to determine minimum passing scores, minimum passing score and the score required to achieve Honors. We hope you will find these results informative.
Assessment of Professional Behaviors Materials Available in DREAM
The NBME is pleased to announce that its Assessment of Professional Behaviors (APB) program is now available through the Directory and Repository of Educational Assessment Measures (DREAM), a peer-reviewed collection of tools within MedEdPORTAL. The APB program materials are available under the Creative Commons Attribution-Noncommercial-Share Alike license, and are thus free for use in the educational setting.
The APB program was developed to support continuous learning around communication and interpersonal skills, professionalism, and practice-based learning and improvement. It is centered on a multisource feedback instrument, designed to be administered using an electronic evaluation system. The materials available for download include assessment instruments, implementation guidance, sample feedback reports, and training modules for raters and for feedback facilitators. A third-party, peer -reviewed critical analysis is included in the download package.
You can access the APB material here. After more than 10 years of development, the NBME is pleased to provide this tool as a service to the educational community.
Data Commons Launches Physician Data Services
Washington, D.C., July 28, 2014 — Data Commons, LLC, a new company that delivers data about physicians, launched its first services today. Data Commons provides data about physician enrollment, licensure and certification-on-demand and directly from multiple source organizations — all through a single hub.
Notice about the Heartbleed Bug
The NBME has been working to assess the impact on our customers in the wake of the recent disclosure of CVE-2014-0160, known colloquially as the "Heartbleed" bug. The Heartbleed bug is a serious vulnerability in the popular OpenSSL encryption software library used to secure the Internet.
Robyn Tamblyn Selected as 2014 Hubbard Award Recipient
The NBME is pleased to announce that Robyn Tamblyn, BScN, MSc, PhD was selected as the recipient of the 2014 John P. Hubbard Award.
Read the full article
Succeed on your career path
Preferred standard for career documentation by the AAMC and NBME
The Pivio® system is a new software tool that will connect data needed by medical students, residents, and physicians across their academic and professional careers, in order to assist users with lifelong learning, career planning, and individual growth.
The Pivio® service is a "secure lock box" that will enable users to control the storage and transfer of data created throughout their academic and professional careers.
NBME Plans Its Centennial
On May 5, 2015, the NBME will complete 100 years of service to the public and the professions.
NBME Executive Board Election
The National Board of Medical Examiners (NBME) is pleased to announce newly elected as well as re-elected members of the National Board of Medical Examiners Board of Directors ("NBME Executive Board"). The election occured at the Annual Meeting of the NBME Membership on April 5, 2013 in Philadelphia.
2014 John P. Hubbard Award: Call for Nominations
The due date for letters of nomination for the 2014 John P. Hubbard Award is September 12, 2013. Information about the Hubbard
Award is found at the link below.
Stemmler Fund Call for Letters of Intent
The due date for the 2013-2014 Call for Letters of Intent is July 1, 2013. An overview of the Letter of Intent application process is provided in the link below.
NBME Partners to Launch Data Commons, LLC
The National Board of Medical Examiners (NBME) and several other leading national physician education and assessment organizations have announced their partnership to launch the Data Commons, LLC. The Data Commons is an information-sharing company that is developing a system to improve the accessibility and use of data about physicians and other healthcare professionals. The core of this new system is a secure data hub that will allow multiple organizations to make available selected information from their individual databases. The five other organizations joining in the partnership are the American Board of Family Medicine, the American Board of Pediatrics, the Association of American Medical Colleges®, the Educational Commission for Foreign Medical Graduates, and the Federation of State Medical Boards. The Data Commons, LLC, is a joint partnership that will administer the new system.
Kevin Eva Selected as 2013 Hubbard Award Recipient
The NBME is pleased to announce that Kevin Eva, PhD was selected as the recipient of the 2013 John P. Hubbard Award.
Read the full article
2012 Fall/Winter NBME Examiner Now Available!
The 2012 Fall/Winter Examiner includes articles on NBME Renewal, the new Global Evaluation Management System (GEMS), and NBME's new online item-writing tutorial.
Read the 2012 Fall/Winter Examiner
International Foundations of Medicine (IFOM) Clinical Science Exam Administered Nationwide in Lebanon
The National Board of Medical Examiners® (NBME®), in conjunction with the Faculties of Medicine at the American University of Beirut, Beirut Arab University, Lebanese University, Holy Spirit University of Kaslik, Université Saint-Joseph de Beyrouth, and University of Balamand, is pleased to announce the first nationwide administration of the International Foundations of Medicine (IFOM) Clinical Science Examination (CSE). Approximately 300 students from these six Lebanese medical faculties participated in the examination, administered on July 14, 2011 at the Lebanese University.
Read the Press Release
2012 Hubbard Award Call for Nominations
The NBME invites nominations for the 2012 John P. Hubbard Award. This award recognizes individuals who have made a significant and sustained contribution to the assessment of professional competency and educational program development at any level along the continuum of medical education and delivery of healthcare.
Learn More About the 2012 Call For Nominations
Lewis R. First Elected Chair of the NBME
We are pleased to announce that Lewis R. First, MD, MS was elected to a two-year term as Chair of the National Board of Medical Examiners at the Annual Meeting of the NBME Membership on April 1, 2011.
Learn More About Dr. First
Judy Shea selected by NBME as recipient of 2011 John P. Hubbard Award
The NBME is pleased to announce that Judy A. Shea, PhD was selected as the recipient of the 2011 John P. Hubbard Award. The award is given to individuals who have made outstanding contributions to the pursuit of excellence in the field of evaluation in medicine. Susan M. Spaulding, a member of the 2011 Hubbard Award Committee, announced Dr. Shea's selection at the Annual Meeting of the NBME Membership on March 31, 2011.
Learn More About Dr. Shea
March 2011 Comprehensive Review of USMLE Update
Since our last update in June 2010, staff has completed successful pilot work in several areas. Test material committees have developed multiple choice questions that assess an examinee's ability to appropriately interpret information presented both in the form of a research abstract and a pharmaceutical advertisement. Additional study is planned, and the items that ultimately appear in the examination may differ from the current design being tested.
Read More About the Comprehensive Review March Update
2010 Fall/Winter NBME Examiner Now Available!
The 2010 Fall/Winter Examiner includes articles on the fall 2010 IAMRA conference and new item formats that are under development.
Read the 2010 Fall/Winter Examiner
NBME @ Events
2019 International Association of Medical Science Educators (IAMSE) Annual Meeting
2019 Association of Standardized Patients Educators (ASPE) Conference
Hyatt Regency Orlando
2019 International Meeting of the Psychometric Society (IMPS)
Pontificia Universidad Católica de Chile
2019 An International Association for Medical Education (AMEE) Conference
Austria Centre Vienna
*As of 2018