Bringing Best Practices and Clinical Tools to Under-resourced Providers - Dr. Rebecca Weintraub, Director of Better Evidence at Ariadne Labs

Hillary Acer:Hi, I'm Hillary Acer welcoming you to Raise the Line with Osmosis from Elsevier, an ongoing exploration about how to improve health and healthcare. 

My guest today, Dr. Rebecca Weintraub, launched the Better Evidence program at Ariadne Labs to design, test and scale data-driven digital tools that help manage diagnostic and therapeutic uncertainty. Currently, Better Evidence engages over 200,000 current and future clinicians and public health leaders in a hundred and forty seven medical schools and clinical sites across a hundred and eighty two countries. 

Dr. Weintraub is also an associate professor in the Department of Global Health and Social Medicine at Harvard Medical School and an associate physician in the division of Global Health Equity at Brigham and Women's Hospital. She's the founding director of the Global Health Delivery Project, a joint initiative between those institutions. 

Today, we're happy to introduce you to Better Evidence, an organization Osmosis has worked with over the years on our joint mission to educate current and future healthcare professionals. Thanks so much for joining us today.

Dr. Rebecca Weintraub:Thank you so much for having me.

Hillary: I'd love to start with learning more about you and what first got you interested in medicine?

Dr. Weintraub:Well, a bit of a long story. I started a social enterprise as a college student called Jumpstart, where we matched college students with Head Start preschoolers who are struggling in the classroom. So really, I guess early on in my career, I had this one-to-one relationship with a young child, and I saw what I could, in service to that child, create. What we were able to basically demonstrate is that children who were not going to enter mainstream kindergarten, with an early intervention, were able to participate in mainstream kindergarten and flourish. 

I was mentored by Edward Ziegler, who was one of the co-founders of Head Start, and while he gave us no money for programming, he did give us a significant resource to begin monitoring and evaluating our services. So, we had longitudinal data from the time a child was three years old and the effects of Jumpstart. 

So, in many ways I brought that current of how can I be of service to an individual, but then how can I make sure I had data at my access to understand the longitudinal impact of a service delivery model?

Hillary: That's fascinating. I actually had a college roommate who worked with Jumpstart and was part of that program, so what a funny connection. Really interesting program. So, yeah, you started out with Jumpstart and you've actually continued working in global health for several years now. What initially got you to focus more on global health?

Dr. Weintraub:Well, I went to Stanford for medical school and we had an incredible ability at Stanford to take rotations outside and off campus from California. So, I planned rotations in Zimbabwe and Ghana and several other countries, and I actually served alongside medical students in those countries. 

I was at the University of Zimbabwe serving as a medical student there, and I saw how hard my fellow colleagues were working on clinical rounds, what type of resources they had, and obviously quite a contrast to being on Stanford's campus.

Hillary: Yeah, there's a big difference there, I'm guessing, between different healthcare settings and systems and what resources they have available. So, when did you actually start to support the healthcare workforce, and what did you notice that they were missing and how did you start to provide those resources?

Dr. Weintraub: Well, I think first fundamentally, I see the agility and the curiosity in all the professionals I've worked with. May they be sitting in Zimbabwe or Peru or Haiti. So, the actual fundamentals of what does it mean to be serving the patient that you're sitting alongside is so similar, obviously. The question is how to bring the same tools, the same piece of software that I use to improve my diagnostic accuracy and my treatment planning that I'm able to have because I have better connectivity and because I have subscriptions to certain tools…can those be scaled? 

In many ways it just became a clinical moment for me. I was serving at a rural site in Rwanda, one of the Partners in Health sites in Rubavu, and I brought DVDs with me, an old laptop -- I mean this is decades ago -- heavy hardware, and everyone was interested. Everyone wanted to functionally see what types of tools were we using, how are we thinking about sub-specialists on call, what evidence were we accessing. That innate curiosity at all hours of the day in all areas of content, I think, really struck a chord in me that there was a way in which we can expand access…that the innate curiosity, the innate agility, the learning interest of the workforce and trainees was universal.

Hillary: I love that you point out the curiosity piece and the lifelong learning that occurs for healthcare professionals because we know how quickly medicine is changing and new research is coming out. So, it sounds like there's always something to be catching up with and staying up to date on. Can you give us an overview of Better Evidence and how it approaches this idea of improving healthcare?

Dr. Weintraub: Well, just to take a step back, we have thought of this as a portfolio approach. What do our clinical leaders need to plan and prepare for the next urgent situation whether they be responding to a pandemic, climate disruption, a change in demographics, or a loss of the workforce. That began actually, in the early days with case studies. We have a library of case studies in global health delivery, which I’m pleased to share. We have a set of tools to help you plan for equitable distribution of a product, and obviously the most immediate is as clinicians, when we leave a patient's room, we have five unanswered questions and we access evidence-based clinical tools, and they have the answers to approximately four of those five questions. 

So, if you are an isolated provider or a trainee where you don't have a learning environment where you can ask questions in the immediate, we believe evidence-based clinical tools help you answer those four questions quickly, improve your clinical confidence, your clinical acumen, and obviously this learning journey. We're trying to help you get motivated, create the habits that this becomes your daily weekly act of educating yourself on the journey to be an excellent clinician.

Hillary: That’s great that you're able to provide more of these resources to clinicians that may not have access and I love the focus on evidence-based science as well. Obviously, that's what Osmosis was built on, was taking some of these research ideas and techniques and actually then implementing them into medical education. Things like spaced repetition or interleaving or the testing effect that of course theoretically we knew about but couldn't always implement. 

So, it sounds like you've taken this big leap from maybe theoretical best practices and research and really found ways to concretely implement them, especially for folks who may not have access. Can you talk a little bit about how this works in practice and maybe some of the impact you've seen in different areas around the globe?

Dr. Weintraub:Sure, and just to build upon your question, I think what we see is many folks -- including clinicians at the front lines and health system leaders -- they actually face a good amount of uncertainty in their daily work. And what we want to do in many ways is get people the best tool early on in their career so they're not going to a Google search. They're going to the best tool that will help them get, in a sense, the expertise that they're stacking alongside the clinical experience or the system level decisions they're making so the next phase of their decision making process is both evidence and expertise based. So, it's a great question. 

What's the impact we've seen today? First I have to say, we have fifteen plus years of testimony from over 200,000 providers saying, “My clinical confidence has improved.” Clinical confidence comes from ‘I know what tool to access in the moment of care. I know what specialists to access.’ People share in many ways the personal isolation they felt prior to accessing the tools. ‘I was unsure, I didn't know who to call. I kept thinking through a clinical case.’ And what we're always concerned about is that isolation affects the ability of that clinician to then learn for the next patient, bring the confidence to be able to say, ‘I don't know’ but hit a safety pause button and find the tool and take that time in either their diagnostic reasoning or in their clinical management for the patient or population they're serving. 

The second is that we've seen people develop into clinical leaders themselves. People feel more and better equipped to be clinical leaders on rounds, for example, to bring the tools even when they're in the intergenerational learning or clinical environment. Maybe their elders have not been exposed to these types of tools before, but they're able to display how they use it and how they integrate it within their decision making process and it becomes, in many ways, a reference point for them to think through, “Wait a minute, I should be asking for these types of tools in the midst of my career arc.” 

And for us, in many ways, the prime example became LexisNexis. We spent some time about fifteen years ago really understanding how LexisNexis had become embedded in legal education. Law students from some of the top law schools in the United States, when they went to the large law firms after their 2L year, actually requested access to it. “While I'm here at your law firm, may I please access LexisNexis? Will you pay for the subscription?” The law firms were like, “Wait a minute, we're not using this in practice.” And once again, you can see the intergenerational work here…the students then requiring this as part of the package that would make them a more equipped, enabled associate. 

We can obviously extrapolate to where we're all learning about large language models today and who has that type of, I'd say, literacy and agility to integrate these tools within their professional roles.

Hillary: That’s fascinating. I didn't realize that LexisNexis -- one of our parent company, Elsevier's, sister companies -- was a driving force behind some of those implementation ideas but we've seen a similar trend obviously with Osmosis where students found us first on YouTube or found our very, very early flashcard app. They found that it was helpful for them in learning and retaining information, and then they ended up going to their academic advisors or faculty saying, “Hey, this is really helpful. We want to integrate this into our education.” And of course there's this network effect that takes off, but it does take time and there's certainly a lot of doubt and pushback in the early stages of that process. 

Did you experience any of that with Better Evidence, and can you talk a little bit about how you and your team overcame that?

Dr. Weintraub:Absolutely. I mean, first it's an intergenerational piece of work, I have to say. I received a PALM Pilot the first day that I started medical school, and I remember looking down at this PALM Pilot and my attendings being like, “Why aren't you paying attention to rounds?” Maybe we were fidgeting, we didn't exactly know how to use it well, but we knew it should be incorporated in some way. And I'm so grateful that Stanford Medical School kind of created that environment. I'm being somewhat facetious. Obviously most of the attendings were interested, “Wait a minute, how do we use this piece for hardware on rounds?” And obviously the software got better, and obviously we all know the story since.

I mean, I personally think what Osmosis has built -- and to be honest, I also use Khan Academy and many others -- is in many ways asynchronously being able to learn even from your peers. That being able to sit in quiet with a device in front of you, being able to rewatch a video, test yourself, self-pace is such a vital part of creating an adult learning process, and it's just been so striking. I think we've had the opportunity to distribute Osmosis to the University of Malawi, and we've gotten just tremendous feedback from our students and faculty, for example. It's very concise and the explanations of complex concepts are easy to follow. I recommend it because the content's comprehensive, it's easy to search and the only wish we get consistently is we wish we had better WIFI or cellular data. 

I'm just mentioning this because I always think of the prerequisites before we introduce a new piece of software. What is functioning, what needs to be in the hands of the individuals? And so in many ways, I think of connectivity as vital as access to water and electricity and I think we're going to see a tremendous uptake in Osmosis and other tools as Starlink is rolled out so significantly in Q3 of this year. So, it's really one to watch. 

Rwanda has access now, and we're seeing people take the hardware where it shouldn't be licensed and starting to turn it on. It’s always a good sign when people are finding and navigating ways, but I think there's just going to be a lot more devices that'll be ready and able to stream Osmosis and many other tools in the near future.

Hillary: I’m glad you pointed out these access issues because -- I don't know if you knew this -- but actually previous to Osmosis, I was at Khan Academy, so I worked with the team there in Mountain View, close to Stanford too, and know that area very well. It's very much a hub of innovation and some brilliant minds. They actually went through this exact same problem several years ago where they developed an app that allowed students to download videos in kind of low data modes and then watch them and access them later on. It was fascinating because, of course, that wasn't the intention of the initial educational videos. Sal Khan was just creating them for his cousin and posting them on YouTube. 

And of course as that globalized, then more and more students around the world were able to access it and then started demanding that they had access to this information. So, I hope that we do continue to see those improvements in access to internet as well as how we manage data, especially large videos that are not necessarily easy to download and things like that. 

In fact, one other note I'll make is we partnered with Wikimedia and Wikipedia a while ago and they had created something very interesting, which is ‘Wikipedia in a Box.’ I'm not sure if you've ever heard of this concept, but they essentially downloaded the entire medical literature of Wikipedia onto a tiny flash drive and USB and were actually able to take that into very rural areas and upload those -- more manually of course -- but it ended up being a huge database of knowledge for groups that may not be able to access that information. So, I think we're on the precipice and we'll see how AI changes all of these things, but hope to continue improving access around the globe.

Dr. Weintraub:I agree. I just did want to plug that we have had an incredible set of interactions with the medical librarians at the medical schools that we work with. We get a chance to work with almost a hundred medical librarians in African medical schools one of whom,Theodosia Adanu from the University of Ghana, won the Osmosis Raise the Line Faculty Award in the librarian category last year. So, I want to first thank Osmosis for this award. Her work in many ways is to be that ambassador, how to explain the value of these tools, how to troubleshoot the connectivity issues, how to help all generations integrate these tools within their clinical or pedagogical practice. 

I think once again, it can’t only be the black box of the hardware and software being delivered, but that local champion who understands the ins and outs of these products and services, who's going to be a longitudinal leader in the settings. What we're seeing right now is a growth in, for example, the medical schools throughout the continent of Africa. More medical schools are opening in the last decade than the previous and so how do we find those champions on the ground? We believe the medical librarians are absolutely key to gain momentum and integrate these tools into practice.

Hillary: I had no idea that you had worked so closely with somebody who won one of the Raise the Line Faculty awards, but congratulations to them. We actually are doing it again this year, so if there are other champions that should be highlighted, definitely keep an eye out for that in the summer. But yeah, we hope to recognize more of these leaders and champions in their different environments because as globalized as we are, it's not always easy to find and recognize them. I guess what I'm trying to point to as well is what's working in those environments and how do you capture those learnings and then share out their best practices across other groups. 

So, was there anything that you've seen some of these champions do on the ground that have really helped to drive adoption, and change the intergenerational usage of some of these new tools and resources?

Dr. Weintraub: Yes. Gosh, I have a host of answers, but I'm going to actually plug one of our Better Evidence champions. I'd love you to interview them so you can hear direct from our medical librarians. Some of them are also the deans of their medical schools faculty and staff. In addition, there's an African Medical Librarian Association, which many of our champions presented at this year and I would say they're using data in many ways. They're able to prove and show their communities that we've been able to turn on more subscriptions. We see what you're clicking on, we see what diseases you're concerned about. 

You can imagine the disease burden for example, that's being faced by the clinical leaders in that community. And I think like any product market fit, they're learning about segmenting users, engaging users. It's just a tremendous skillset beyond marketing, beyond behavior change because they themselves understand how hard it is to be a medical school student or a faculty member in the environment they're in. They understand what it means to not have continuous connectivity. They understand, to be honest, just the burdens and responsibilities that so many of the students and faculty face beyond the learning environment.

Hillary: Well, I hope we do get to interview more of them. We'd love to feature them on the podcast and hopefully more of them get recognized in the faculty awards that are coming up as well. 

You have also advised a number of ministers of health, so not just medical librarians, but government officials as well and health systems leaders. How do they integrate this work and are there any examples that you've seen in different countries, or even broader groups, that have built or implemented these systems at scale?

Dr. Weintraub: Great question. So, first I have to say most new ministers of health are clinicians by training, so it tends to be a physician who's politically appointed by a prime minister and has basically twelve hours to be prepared to become the minister of health of their country. That means while they have tremendous medical training in a subspecialty, maybe OBGYN or a neurosurgeon or orthopedics, remember the bulk of their first few decades of their career has been their clinical acumen, they’ve developed their diagnostic reasoning. 

That's quite different than managing the uncertainty of financing a health system, of managing multilateral donors, of thinking through and planning for the equitable distribution of scarce resources, whether it be a new diagnostic or a vaccine. And in many ways, what we find some of our ministers of health need is to think about how am I going to communicate with my minister of finance? How am I going to explain the return on investment of expanding health access, for example, of ensuring there's community-based organizations that are integrating within my primary care services, et cetera. You can think through that complexity. I think a lot of the clinicians who are listening could think about what would that feel like if I had twelve hours to prepare. 

So, we want to help you prepare in that moment, and absolutely there's ministers of health that we support, and also ten years before you become the minister of health. We've been running a program for fifteen years and several of our graduates have become the ministers of health of their countries, and we want to help prepare you with understanding what it means to be the decision maker, manage uncertainty, tools for equity planning, understanding your supply chain, health financing, et cetera, on top of your clinical background, your clinical experience that makes you such a tremendous leader.

Hillary: I can only imagine what it's like to go from being a clinician, maybe running smaller programs or some programs within your clinical settings, but then going to managing an entire country. That's quite a big jump. Can you speak a little bit more to the program that you're running and what skills and competencies your leaders are going through?

Dr. Weintraub: Sure, absolutely. We run a summer program called the Global Health Delivery Intensive. It's a credit-bearing activity between Harvard Medical School and Harvard School of Public Health and this is very translational and practical. We're trying to help you understand aspects of these jobs you will likely encounter. You'll be supervising an epidemiologist, you'll be supervising biostatisticians…how do you read the papers and then plan the policy translation. We have a course on management practices and global health delivery. How do we help you build robust information systems so you have the information you need at time? What are the research questions you need when you're planning for scarcity? For example, how do you combat and think about the effects of climate change? Maybe nutritional deficiencies, for example? And of course technology is a large thread of this…which are the pieces of software that'll help you plan and prepare for your role. 

And as we know, in most of these public facing roles, the question is not how do I support a population? It's the entire population. So equity is first and foremost, the roles and responsibilities of a new minister of health or also someone who's serving as, for example, the head of laboratory capacity, the head of pandemic preparedness for our country.

Hillary: Fascinating. It goes back to what you said about how much uncertainty not only clinicians are facing, but also government officials, ministers of health. It sounds like you're really building this capacity to think around building systems for equitable distribution and trying to make sure that you're meeting the needs of a very diverse population. I would love to follow that program a little bit more. I've been dabbling a bit in public health again after my previous education in biology, and a lot of these things are coming back to me as I am talking to you. So, very interested to follow the programs and see more of the impact. 

Just to close out…we are obviously an education company. One of the things that we try to do is to bridge knowledge gaps and to make sure that we're also increasing access on topics that are very important. Is there a topic that you think Osmosis should create educational materials around that you think would really help the population that you're serving?

Dr. Weintraub: I think this is actually for all of us as providers…the iteration we all need to do in our own personal practice is building trust with our patients. That changes over time depending on whom you're serving, what generation, what your background is and having almost that 360 degree review of yourself. Am I communicating in a way that the patient in front of me understands the language that I'm using or the complexity of what I’m talking about? If I'm presenting a differential diagnosis directly to a patient, or I'm giving them the results of a diagnostic test, or I'm defining a treatment plan, how am I going to communicate this? 

But most importantly, we need that type of evidence-based feedback. What does it look like to build trust? Am I establishing good practices? Obviously being part of Ariadne Labs, the checklist has been very much a practice that we think through, and I think we need a sense of pedagogy regarding trust building with patients and being able to check yourself, gosh, have I communicated in a way and have I checked that the patient understands and can report back on what I've just said? So, I still think most of the work is the interchange, the exchange, and really being to meet the patient and make sure we're gaining understanding in the course of a clinical encounter.

Hillary: Yeah, wow. I think it takes a lot of vulnerability and maybe courage to be able to receive that feedback and integrate it and can just imagine a lot of the pressures that clinicians are under to deliver positive health outcomes and to make sure that they're meeting their patient's needs. But then also on the flip side, to take that time to evaluate how they've interacted with them or if they've built that trust. 

So, can you share just a little bit about the checklist you're talking about or what you've seen really work with trust building between clinicians and the patients that they're serving?

Dr. Weintraub: Sure. In my own practice, I've integrated the serious illness checklist. We used to print out a copy of the serious illness checklist. You'd have a paper copy that you bring to keep reminding you to follow the order. It was so well designed and studied and so similar to being in the cockpit of an airplane. For example, the idea was I didn't need to worry about the orders already defined, but if I followed this order, I would have a comprehensive and concise conversation with the family to really understand what a good day looks like, what their wishes and desires are. This is modeled. I think many listeners will know of the relationship between copilots and the cockpit. 

I think we can have structured forms and conversations for trust building for these notions that we think of as broad and cultural, but there's a way in which with some structure and patterns, it helps the clinician make sure they're being both comprehensive and their practice is improving because it's less at the end of the conversation, but it's part of the engagement and how you're framing the whole encounter with patients and families.

Hillary: It's interesting. We had another guest -- Dr. Steven Halm, the Dean of Des Moines University College of Osteopathic Medicine -- who mentioned this connection between aviation and some of the learnings and safety practices within aviation and how they've pulled over to medicine. So, it’s interesting that you're citing that as well as something that can potentially be helpful with not just safety guidelines, but also trust building. So, I'd love to see more of that implemented since it's clearly been effective. 

One other question we'd like to ask is for our audience of students and early career health professionals, what advice do you have to them about meeting the variety of challenges that they will likely face in healthcare in the near future?

Dr. Weintraub: It’s a beautiful question. One of my clinical mentors asked me to become fascinated by one drug or one diagnostic and follow it through our health system. I became fascinated by argatroban, and my advice is similar - to be curious about one diagnostic or one drug. Understand the chemistry, the pharmacokinetics, how it's administered, how it's priced, and think through the relationship, the manufacturer, the regulator, the provider, and most important, the effects on patients. How do they integrate the introduction of a new product into their daily life? What does it mean to have to pick up a new prescription? Or if they're receiving an infusion, what’s the complexity of integrating that alongside all their other roles and responsibilities?

I mention that because it's really in many ways to just be so curious and take it through maybe one diagnostic or one molecule, but then, gosh, you start functionally understanding complexity not as a complicated diagram in many ways, but you start seeing the complexity of navigation. And I think that's a large part of our role as clinicians, is to help patients navigate with less stress, better evidence, and more understanding of how hard it is to be a patient and the work that the patients are doing to both navigate the system, manage their stress, and the uncertainty to have better health outcomes.

Hillary: That’s really interesting advice and would love to see maybe how that has worked in practice for you, too. I can imagine what diving into a specific problem like that or specific drug and following all of the details of that, including patient adoption of that drug would teach somebody in that journey. So, really fascinating and I really enjoyed our conversation today, Dr. Weintraub. Is there anything else you would like to leave our audience with?

Dr. Weintraub: Thank you. I do think the most poignant parts of my career have been moments of direct service. I enjoy teaching, I enjoy mentoring, and I think it always has to be this robust, active combination of seeing the next generation be well-equipped, and having your own moments of service, whatever that direct service may be for you as a professional, but that keep you very centered on why are we here, whom are we serving, and what are their needs?

Hillary: I love that reflection. That's a really, really great note to end on. Thank you again for being with us today.

Dr. Weintraub: Thank you so much for having me.

Hillary: I’m Hillary Acer. Thanks for checking out today's show. Remember to do your part to raise the line and strengthen the healthcare system. We're all in this together.