Stemmler Fund 25th Anniversary Interview Series
The Stemmler Fund is integral to NBME Contributions, which celebrate and support the medical education community. On the 25th of every month, you can learn about a different Stemmler recipient's journey and how this research opportunity shaped and influenced their career in medical education.
Kevin Eva, PhD
Kevin Eva, PhD, received NBME Stemmler Fund grants as a principal investigator in 2003 and 2011. The 2003 grant helped develop his research into the Multiple Mini-Interview (MMI), a medical school admissions tool that utilizes scenarios and standardized interactions to assess interpersonal skills and professionalism. In 2011, his research promoted better understanding of rater error for performance-based assessments by exploring underlying cognitive processes.
In addition to serving as principal investigator on these projects, he was also a team member on Stemmler Grants awarded in 2006 and 2010.
Recently, NBME caught up with Dr. Eva to discuss his career, his research and the medical education community in celebration of the Stemmler Fund’s 25th Anniversary.
How did receiving a Stemmler Grant influence your career?
The Stemmler Grant provided the resources to engage in true exploration. The prestige of the award and its highly competitive nature contributed to the project’s credibility and drew more attention to it.
How does the Stemmler Fund help medical educators overcome challenges they face in their profession?
The most typical barriers that you hear about when people in our field try to pursue research are time, money and expertise. When you have the opportunity to do research financially supported by Stemmler, it also creates a tangible activity that people with complementary expertise can rally around.
The benefits of collaboration are hard to realize until there is something to actually collaborate on. For me, the Stemmler opportunity consolidated a group that was just emerging with a special interest in medical school admissions. It gave us something specific and concrete that we could work on together that wound up being a years’ long collaboration.
How do you think the Stemmler Fund adds value to the medical education community?
There’s not a lot of funding available for education and assessment research. The Stemmler Fund reinforces that this type of work is valuable and drives the field forward in terms of modernizing education practices. When administrators and leaders see that there is an organization like NBME who will financially support ways to modernize assessment–advancing both thinking and practice–I think that leads them to take notice of the work’s value.
The purpose of the Stemmler Fund is to foster innovation in the assessment of medical education. Why is the assessment of medical professionals important?
It’s one of the most crucial things we do. As a profession, medicine needs to take the obligation of assessment very seriously to ensure protection of patients and society. Assessment is an invaluable part of the quality assurance and regulation mechanisms that are built into that activity.
What advice do you have for an individual submitting a proposal for a Stemmler Grant?
Work collectively as a group to do your homework and lay the groundwork for why your idea is interesting, important and novel. Take the time to nurture the idea and explore what others have done that might be similar. Make sure you can identify and clearly state what the actual problem is that your work is going to help solve, and how your project builds upon the work that others have done. Make sure you have the right people around you to challenge you in developing ideas and to bring their complementary expertise together so the collaboration makes the work better.
Once the proposal is written, I would encourage one to submit it to peer review before they submit their application to the Stemmler Fund. Make sure you get the advice of trusted and informed colleagues who will tell you where the gaps in your thinking lie, because we all have our own blind spots.
Is there anything else you would like to add?
Just ongoing enthusiasm for and congratulations to NBME’s efforts to support the education and assessment communities through the Stemmler Fund. The fact that NBME has been able to keep this going and use this as an opportunity to reinvest in the field provides invaluable opportunity and ensures much greater impact than would otherwise be achievable.
Patricia O’Sullivan, EdD
Patricia O’Sullivan, EdD, received an NBME Stemmler Fund grant in 1998 to research the use of portfolios in medical education. Her innovative project demonstrated how to develop and provide validity evidence for portfolios as an assessment for residents. Her grant provided support to further study this once underexplored area.
Recently, NBME caught up with Dr. O’Sullivan to discuss her career, her research and the medical education community in celebration of the Stemmler Fund’s 25th Anniversary.
How did receiving a Stemmler Grant influence your career?
Receiving the Stemmler was pivotal to my career. The grant gave me the opportunity to carry out a series of research studies that allowed me to contribute to the understanding of the use of portfolios in medical education. That led me to consulting with other groups interested in implementing portfolios and to continue to write in this area. The grant allowed me to develop expertise in an area that would not have happened without that support. Support was not only in the funding but the recognition that this was a worthy area to pursue.
How does the Stemmler Fund help medical educators overcome challenges they face in their profession?
There are relatively few grant opportunities focused specifically on medical education and Stemmler is one of the best known. The effort put into developing the proposal and feedback received position the recipient to begin a program of research studies. Such a program is necessary for career success and also provides valuable information to the community. Demonstrating that one can obtain funding for educational research is seen as a sign of being a scholarly faculty member and enhances one’s career. In my own case, the funding allowed for consultancy, which enhanced the quality of the process, and it allowed us to fund the needed studies.
How do you think the Stemmler Fund adds value to the medical education community?
The Stemmler Fund is highly regarded as a source of funding for innovative ideas in assessment in medical education. While it advances the scholarship of individuals and teams, the work has influence on the practice of medical education.
First, the work is disseminated. Medical education is a relatively small community and the outcomes from Stemmler funded projects are read widely. Second, the studies often have contributed to the evidence for good educational practice.
The purpose of the Stemmler Fund is to foster innovation in the assessment of medical education. Why is the assessment of medical professionals important?
For a long time, much of assessment was framed in the concept of accountability. We needed assessment to demonstrate that individuals had the knowledge, skills and attitudes of a clinician that the public can trust. While this is important, there has been a shift toward assessment for learning, where the assessments provide feedback that can be used in a timely manner to help individuals improve. From both perspectives, assessment is essential to provide a quality practitioner.
What advice do you have for an individual submitting a proposal for a Stemmler Grant?
Many individuals have an idea of what they would like to propose. My advice is to fully develop the proposal as best one can even before submitting the LOI [Letter of Inquiry] since this allows the individual to clearly convey the gap that they are addressing and the methods that they will use. I advocate using internal peer-review prior to submitting to make sure that the proposal is clear. I encourage submitters to make sure that the importance of the contribution of every team member is clear.
Is there anything else you would like to add?
Having a Stemmler award is something that I am still very proud of several decades later. I appreciate the effort made to fund quality, innovative ideas and the fact that Stemmler fills a gap in the funding landscape because of its focus on education and, specifically, assessment.
Larry D. Gruppen, PhD
Larry D. Gruppen, PhD, received an NBME Stemmler Fund grant in 2003 to research self-assessments. His innovative project gathered data on a group of learners over time to assess individual differences in self-assessment accuracy.
Recently, NBME caught up with Dr. Gruppen to discuss his career, his research and the medical education community in celebration of the Stemmler Fund’s 25th Anniversary.
How did receiving a Stemmler Grant influence your career?
It had a number of significant impacts on my career. One was the opportunity to pursue a program of study that had an ongoing focus on self-assessment and various related assessment issues. The Stemmler Grant also provided a focus for a team of collaborators who supported and stimulated each other’s thinking about self-assessment.
How does the Stemmler Fund help medical educators overcome challenges they face in their profession?
Medical education researchers tend to focus on local, institutional problems and seldom have time for taking a broader perspective on fundamental educational or assessment issues. The funding and prestige of the Stemmler Grant helps scholars protect and devote effort to these larger issues. As one of the few designated sources of funding for medical education research, it provides credibility for a career focused on medical education scholarship.
How do you think the Stemmler Fund adds value to the medical education community?
The Stemmler Fund focuses attention on the importance and the rigor of assessment in medical education. It encourages both innovative exploration of key educational outcomes and meaningful methods for assessing those outcomes. Both are necessary if medical education is going to fulfill its obligation to provide competent health care to patients and families.
The purpose of the Stemmler Fund is to foster innovation in the assessment of medical education. Why is the assessment of medical professionals important?
To me, meaningful and trustworthy assessments of physician performance and educational relevance are key to the social contract society has with the medical profession. The medical education community must be able to trust physicians but also make decisions about resource allocations, regulatory guidance, and individual performance. The Stemmler Fund’s emphasis on improving assessment in medical education contributes substantially to developing assessments for skills and performance that were often assumed. It also helps the discipline adapt to changes in the practice environment, such as the growing importance of teams and systems of care in patient outcomes.
What advice do you have for an individual submitting a proposal for a Stemmler Grant?
Because of the assessment focus of the Fund, applicants should seek out colleagues with technical experience in psychometrics and quantitative methods, as well as advice from mentors and consultants with assessment expertise.
David A. Asch, MD, MBA
David A. Asch, MD, MBA, received an NBME Stemmler Fund grant in 2006 to research assessing the quality of clinical training programs through measuring the clinical outcomes of patients cared for by the graduates of those programs. His innovative project successfully evaluated training programs through this methodology and connected the assessment of medical professionals to what patients most value: the success and quality of the care that they receive from their doctors.
Recently, NBME caught up with Dr. Asch to discuss his career, his research and the medical education community in celebration of the Stemmler Fund’s 25th Anniversary.
How did receiving a Stemmler Grant influence your career?
The Stemmler Grant was used in an incredibly leveraged way to get a piece of work done related to the measurement of medical education. It turned out to be more influential than we ever would have imagined, and it ended up creating a subtheme in my own research career that never would have existed. And so, for me, it was transformative.
Could you provide an overview of your research project that was funded by Stemmler?
I came to the project with a set of ideas developed with my colleague, Sean Nicholson, PhD, about how we would assess medical education. It was based on the question, “How would you know that a training program was any good?” The answer we came up with is that we could judge a program as “good” depending on whether it produced doctors who produced good outcomes for the patients. We took the measurement of medical education quality right to the patient. That turned out to be a novel concept and one that people could understand. Then, we demonstrated that it could be done. I think that single concept and its execution ended up resonating with the world of medical educators.
Reflecting on the original purpose of the Stemmler Fund that was established 25 years ago, why is the assessment of medical professionals still important today?
In some respects, that’s a very easy question to answer. What would the world be like if we didn’t have assessment? Why would we even want to go to the doctor if we didn’t have some kind of assessment of educational process? That’s the simplest answer I can provide; we would hate not to have it.
How do you think the Stemmler Fund adds value to the medical education community?
When you create a funding program, you’re giving validation to an area. You’re saying this is something that’s worthy of investment, worthy of attention, and worthy of people’s effort. That gives people motivation and inspiration to believe that a field is worth their time.
Is there anything else you would like to add?
I think the Stemmler Fund serves the field of medical education incredibly well. First, it honors a man—an unassuming, quiet and yet visionary leader—who made important contributions to American medical education. And second, it offers support for those new investigators who can advance medical education further. I’m glad that the Fund still exists, and that it’s helping new generations of people make contributions to a field that we never want to go away.
Maxine A. Papadakis, MD
Maxine A. Papadakis, MD, Professor of Medicine, Emeritus, at the University of California, San Francisco received an NBME Stemmler Fund grant in June 2002 to research professional behaviors of individuals in medical school and throughout their careers. Her groundbreaking project concluded that a person’s unprofessional behavior in medical school was associated with subsequent disciplinary action by a state medical board.
Recently, NBME caught up with Dr. Papadakis to discuss her career and the future of medical education in celebration of the Stemmler Fund’s 25th Anniversary.
How did receiving a Stemmler Grant influence your career?
The Stemmler Grant completely changed my career. It allowed me to look at the field of professionalism through a unique research lens. Because of the Grant, I have been able to remain in the field of professionalism research. It has been enormously satisfying to be in the field as it expanded from a smaller discipline to a larger one.
It is a framework for students as they come across different scenarios of professional judgment and action. There is no way that someone can be trained about all the specifics that they will encounter in future situations. But medical providers have to be trained to recognize, to analyze and to have a foundational underpinning for what they will face.
How do you think Stemmler adds value to the medical education community?
Initially, the Stemmler Fund was almost the only funding source for the study of professionalism. It remains a critical source that funds core innovations in professionalism. Inventive, out-of-the box research supports the discipline of professionalism as a core competence and enables researchers to take risks in opening new areas of investigation. Especially in these times, we need to celebrate the extraordinary selflessness of our health care providers and learn from their professionalism.
What advice do you have for an individual submitting a proposal for a Stemmler Grant?
As with any grant proposal, it is important to carve out something specific, measurable and doable. It is also important to stay realistic about what can be done, and to stay relevant to the struggles of learners and patients today.
Is there anything else you’d like to add?
I am grateful to Stemmler and to NBME for housing the program. Twenty-five years ago, when we started to study professionalism, there wasn’t much happening in the field. Now, professionalism is truly a core competence for learners, and the study of outcomes related to professionalism is front and center. The Stemmler Fund has been instrumental in making that happen.