How AI is Shaping the Work of Medical Educators - Dr. Andrew Rodman, Co-Director of iMED at the Beth Israel Deaconess Medical Center

Shiv Gaglani: Hi, I'm Shiv Gaglani. Today's Raise the Line guest is right up our alley at Osmosis, given his interest in using digital technology and artificial intelligence to improve medical education. Dr. Adam Rodman pursues that goal as co-director of the Innovations in Media and Education Delivery Initiative (iMED) at the Beth Israel Deaconess Medical Center, as well as being co-director of the digital education track of the internal medicine residency there.

In another similarity, Dr. Rodman also hosts the Bedside Rounds podcast -- which is wonderful -- which he produces in partnership with the American College of Physicians. We were first put in touch by our mutual friend, Scott Carney, who we had on the podcast, and both of us are fans of the books he's written.

Adam, it's a real pleasure to have you on. Thanks for taking the time to join us.

Dr. Adam Rodman: I don't know that I could ever possibly follow Scott, but I will try.

Shiv: (laughs) We always like to ask our guests to, in their own words, tell us what first got them interested in medicine, and then in your case, internal medicine.

Dr. Rodman: That's a wonderful question, and it's a difficult question to answer because we always have a tendency to look back and tell stories about ourselves that may or may not have been true.  Actually, like you, I am what was called, or still is called, a non-traditional applicant to medical school. I worked for an economic policy think tank after undergrad, actually for almost three years, doing microfinance stuff. This is when microlending was really popular.  My intent had always been to actually go to grad school and get a PhD. I just hated sitting at a computer and analyzing data all day and not talking to anybody. Then I made the kind of crazy decision to go to medical school, and I did not have all the prerequisites. I didn't know that. I was very stupid. I didn't know that post-bac programs existed, so I just took organic chemistry in the labs while I was working full-time.  If anyone is listening, I highly recommend going against that.  

Then as for internal medicine, I could never make up my mind. I'm still a general internist to this day. I work in the clinic, and I work as a hospitalist, so I work everywhere. I liked everything, and I could never make up my mind about what I wanted to do, and I guess I still haven't.  Internal medicine was a natural fit. You're, like, “I'll wait and see what I like,” and now over a decade later, here I am, a general internist.

Shiv: I love that. Lots of shared similarities, I think, in being non-traditional.  At what age were you when you started med school and then finished?

Dr. Rodman: I was twenty-seven. I think that's pretty normal these days.

Shiv: Yeah, I think so. We had Lisa Sanders on the podcast, who you may know from Yale.  She started after a career in TV journalism at thirty-seven at Yale. When I started med school the first time around, I had a classmate who was forty-one who was an immigrant electrician for many years and then wound up going. I like this. It seems to be normalized...stories like yours and mine and others.

Dr. Rodman: It's a variety of experience. I think bringing in a variety of experiences outside of the strict confines of medicine is good for the field.

Shiv: Totally. Moving into the work with iMED, tell us about what got you interested in academic medicine, medical education, and then ultimately co-directing iMED.

Dr. Rodman: I've always questioned why we do things and the big ‘why.’ Why do different medical practices exist? When I was an intern, I found that the ‘why’ part of my curiosity wasn't really being scratched very much. I was mostly writing notes and being told to do things, even when I thought they were really stupid and didn't have much justification.  

One of the things that really got me through my intern years is a podcast. This is a long time ago. In 2013, medical podcasts were not really a thing yet, but I became obsessed with Radiolab. I was already obsessed with Radiolab.  This is when Jad Abumrad and Robert Krulwich were hosting. Lulu Miller and Latif Nasser, who are also amazing, are the hosts now.  Back then it was Jad and Robert. They had this blend of storytelling and very real science and its connections to the larger humanity that showed me a way forward with science or medicine... that it didn't just have to be rote, doing the same things, doing as you were told, tradition being handed down. 

Those were my inspirations, actually, when I was a second-year resident, to start my own podcast and to think about digital education. It was not called digital education, actually. Back then, the term FOAM (Free Open Access Education) was still used, but FOAM was just bubbling up which is this idea that medical education could be asynchronous, but also decentralized and away from the traditional top-down hierarchies. Again, I wasn't coherently thinking about this. It's a story I'm telling myself now that my hair is gray, I'm getting old.  That's what I was clawing towards. 

When I started Bedside Rounds, I really started it for myself, not as part of a grand plan to explore these ideas. As I transitioned, I actually did a fellowship in global health, worked in Botswana in medical education and then came back to the United States. I started to realize the importance and power of digital education, that some of these inchoate longings that I'd had for a medical education that was more democratic, more decentralized, that took in different perspectives and was actually effective at calling bullshit effectively on things we do for no reason, saying, “Oh, some of these things we're doing are the way we think and don't make sense. These are just tradition.” 

I should say that's what Osmosis is also. I actually read a paper you wrote that was in Academic Medicineabout the global reach of Osmosis. That was pretty inspiring.

Shiv: Thank you. 

Dr. Rodman: Showing the power that these digital platforms and digital education had. Me and Shreya Trivedi actually started iMED in 2019 or 2020 -- the pandemic makes my brain fuzzy -- as a research organization and an advocacy organization to study and advocate for and train residents and faculty members in digital education.As all of this is happening, just as with the work that you've been doing, digital education has taken over. It doesn't even make sense to speak of it as separate from medical education anymore. In some of our own research and others' research, it is the way that people learn now.It's the way that our residents learn.  If you look at emergency medicine attendings and APPs, it is the dominant way that they learn.  Digital education is now synonymous with medical education. We were talking about this a little earlier. 

What's really exciting and scary in medical education right now is that we're seeing huge artificial intelligence, large language models enter the scene.It will definitely be a disruption to medical education.  I know our medical students are already using it. Just as digital education took over a decade -- and I won't even say it's mainstream -- but a decade to develop, we're seeing these changes happening very rapidly in medical education, much faster than, say, podcasts or YouTube videos for medical education.It's been a really exciting and whirlwind time to be a medical educator.

Shiv: Yeah, I know. Totally.  I actually didn't know about the global health connection with Botswana. Let's dive into that a bit.  We'll definitely have more time to talk about AI and LLMs.  That's one of the main reasons I reached out to get you on the podcast. I was actually just in Rwanda and Tanzania back in February visiting a school that we worked with there, the University of Global Health Equity. Personally, you may know from my background, I was born in Namibia, so sub-Saharan Africa is very near and dear to my heart. 

Part of the promise for Osmosis, the reason we developed it, was the vision is to educate a billion people by 2025, and that includes people who traditionally would not have access to medical education because they can't get to the lecture halls in Baltimore or Boston, or they can't afford the books. But because the marginal cost of delivering an online video, especially on a platform like YouTube – thank you, Google, for having all of our hosting costs taken care of – is so low that you can get these resources to as many people as possible.  You can crowdsource. You can learn how they do it. Frankly, a Botswana med student, I think, is much better at treating certain tropical diseases than any U.S. med student could ever be just because they're so close to those issues.  Tell us a bit more about the global health aspect.  Are you still doing anything in global health? 

Dr. Rodman: Well research-wise, but I don't work in Botswana anymore. This will be a plug for anyone applying to internal medicine. Beth Israel Deaconess Medical Center has long had a partnership with Scottish Livingstone Hospital and the University of Botswana via the Boston-Harvard partnership, so a lot of HIV-AIDS research. Oh my goodness, for seven or eight years -- it might even be a decade now -- we've been having an exchange. We send our residents to Botswana.  I was one of the faculty members, so I was a fellow, but I was the internal medicine attending at Scottish Livingstone Hospital.  

It's been about a decade now, but they launched their own internship training program. In the past, they had relied on neighboring countries and European countries for their training, but now they have their own internship program. It was a really exciting experience, a humbling experience from the very beginning, thinking about what an internship curriculum looks like and what evaluation looks like in a country like Botswana and building that from the ground up.

One of the reasons I'm passionate about digital education is the opportunity to, as you say, educate, but to widely make a lot of this didactics training -- that has been kept in ivory towers or really siloed away -- available across the world. Some of my research now is on the difficulties. You need to engage with local partners, you need to bring in local expertise.  It's a lot harder than just saying that, and the opportunity is still there and it's partially realized. I think there's a lot of opportunities to improve global medical education going forward.

Shiv: Yeah, absolutely. Very cool.  Do you know Dr. Sean Tackett?  I'll make this introduction later.  We're working on an AI grant for Hopkins. Recently, he got an award from the Provost’s Office, but he also has done a lot of work in General Internal Medicine at Bayview Hospital at Hopkins and does a lot of international medical education work, too. So, anyways, I'll make that connection later.

Dr. Rodman: Well, thank you.  This is very nice to do on air.

Shiv: I was just on the call with him earlier today about the grant.  So, before we go into AI again, let me just pull on Bedside Rounds. It’s a great podcast.  You cover such interesting and diverse group of topics in the history of medicine. Maybe tell our audience who may not be familiar with it what you've been covering, how you choose topics, and what the goal is of Bedside Rounds.

Dr. Rodman: Yeah, I'd say the goal is explicitly didactic. It is a storytelling podcast in the style of Radiolab, but the goal explicitly is to explain how modern medicine became the way that it is and focusing a lot on thought, on epistemology, on what it means to think and what it means to be a doctor. 

The answer to how do I choose things is that the topics intersect with my professional life.  I've been doing a lot of things lately on the history of diagnosis and when I say that, I mean diagnosis as an idea.  What does it mean to make a diagnosis? How has our conception of diagnosis changed over time -- and it's changed a ton -- and how does it continue to change? That has led into a lot of topics that are related to a lot of things that matter a lot to us, like the structure of electronic health records, how we interview our patients, the relationship between doctors and patients. 

It's also led into a lot about clinical decision support, the history of that, and artificial intelligence.  Though as a history podcast, I think the closest that I've ever gotten to like the present is 1979. So, it is still firmly based in history.

Shiv: I definitely like listening to the podcast. I can tell why you and Scott Carney are friends...just being super motivated by your curiosity and finding what others would consider super esoteric, but finding really interesting threads.

Dr. Rodman: Esoteric is a good word.

Shiv: But insightful and interesting.  What's like a favorite episode that you want to share with our listeners. Something that you enjoy?

Dr. Rodman: That's such a good question.  An older episode that I really, really enjoy is called The Cursed. I made this like five years ago.  It is a famous story about Charles II of Spain who was famously inbred and he died at a very young age. His autopsy famously showed him, and this is quote, “a heart the size of a peppercorn, a head full of water, and a bloodless body.” The episode does not explain what those pathophysiological findings are, but what it does is it looks at like Michel Foucault and this idea of the clinical gaze and how our conceptions of where disease lives in the body affects what we see and how we approach our patients. 

So, it's an older episode, but it's a really fun anecdote and it gets into like Foucault and the birth of the clinic.  It was very important in a period of medical history. So, it really gets into some of these meta-historiographic trends from the sixties and seventies and eighties in a way that I think is fun.

Shiv: It sounds like a lot of fun.  I definitely want to go back and look at episode forty-three and go back in the archives. One of my favorite classes in undergrad that you may appreciate because you're in Boston was a history of medicine class. My paper was on the assassination of James Garfield. The bullet was lodged in his spinal cord and there were all these great physicists, including Alexander Graham Bell, who were called in to see if they could detect where the bullet was because they couldn't find the bullet. So, there were actually early versions of devices they had created that were still in Countway Medical Library and it's kind of cool to actually touch the device that was used on a president.

Dr. Rodman: Yeah, he used like an inductive machine, effectively, a metal detector and arguably the first medical imaging ever done in history.  It's quite remarkable.

Shiv: Totally. Yeah. So, it sounds esoteric, but it's like very cool when you actually dive in.

And recently when I was on my London trip, I visited the University of Edinburgh. I had no idea there was so much medical history out of Scotland. 

Dr. Rodman: Edinburgh is one of those centers of medical history in the 19th century. Yeah. 

Shiv: I mean, Arthur Conan Doyle went there and he based Sherlock Holmes on his professor. I think it was James Bell or one of these great physicians who, you know, could just look at somebody and have like an intuition around what they do and we can just think about Sherlock Holmes as another physician.

So, anyway, we could take the rest of this show talking about medical history and these interesting things, but let's go into AI. Most recently, I really enjoyed reading your perspective in the New England Journal of Medicine, which published an article you co-authored in early August called “AI and Medical Education: A 21st century Pandora's Box.” Our mutual connection, Dr. Eric Topol, highlighted this and spoke very highly of your perspective on it on LinkedIn.  For our audience who may not have read it yet, what are some of the things you're thinking that either keep you up at night out of out of either being fearful or being really excited about AI in medical education?

Dr. Rodman: And to be clear, I'm both. Actually at the end of the piece -- I this is very postmodern -- I quote Jacques Derrida to bring up this idea of a ‘pharmakon’ because in ancient Greece, the word for poison was the same word for medicine. So AI, I think, has both great potentials for med ed and great fears.

The gist of the piece is that -- and I think you're like this, too -- in the clinical reasoning community, I'd say there's kind of a split between people who take AI seriously and people who think it's a fad. I take AI very seriously and the reason I take it seriously is I’ve used it myself, but also studied it. Me and Zahir Kanjee and Byron Crowe ran all the New England Journal CPCs through it and actually does make diagnoses very well and I'm running or helping to run a number of randomized control trials. It's not perfect. It cannot replace a doctor right now, but it truly can show like remarkable insights into the ways that we understand how humans think. 

So, my reason for writing this piece is that I was very frustrated and scared because I saw really the discourse going two ways. One way was just ignoring AI...medical educators saying, “I don't know what this is.  This is probably nothing. It's hype from the tech community.  Everyone was excited about blockchains and none of that has come to pass. This is just yet another tech bro hype thing.” And then on the other side, I saw people, taking AI seriously, I guess, but focusing really on short term and very real challenges such as hallucinations, such as the propensity to make up information. That’s a very big deal in medicine, such as bias in the training data. 

We were talking about the global health medical education.  Well, LLMs are largely trained on data from high-income countries with the majority using English and they reflect a lot of the same biases that we have, including racial and sex biases. So, very real problems. But I saw the focus either being dismissal or focusing on these very immediate issues and as a historian and as someone very interested in what it takes to train the medical mind, I was a little bit more worried about medium-term views or the medium-term effects on medical education. 

A lot of this was compounded by talking to my own medical students about how they were using LLMs already and so the gist of my worry is -- and you'll agree with this -- is that while medical education is imperfect in very many ways and it's hampered by tradition, we still have a pretty decent sense of how to train doctors. A lot of it is strip away the lectures and stuff. A lot of it is practice. It's supervised practice. That's what residency is. That's what the clinical wards are. It's seeing patients over and over again, building up mental models of diseases and patients.  We call those illness scripts, or we call those schema, depending on exactly what mental tasks we're doing, and then like training when we switch to other mental heuristics and then getting feedback over time, right? That the cognitive apprenticeship model in a nutshell. 

Now, LLMs are very different than the EHR given the ability to offset or to take a load off some of those mental processes. There are medical students right now using LLMs from day one of medical school to help. Not just like how you would use Google to get a fact, but to use reasoning processes to make diagnoses to help them work through their cases. In some ways, this is very good. 

I spoke with one medical student who is actually, like us, a non-traditional applicant and had a lot of imposter syndrome. Before his PBLs, he was treating ChatGPT as a tutor, giving it know, pieces of the PBL and getting feedback and asking questions. So, there are some very, very positive things about this also.  But we're now dealing with medical students from a very early age offloading some of these cognitive processes and these are like circa 2023 LLMs, so I can only imagine that these things are going to improve over time.  

Medical education needs to start to grapple with what does it mean for med ed when a lot of these cognitive processes that medical education is designed to train for get offloaded to a machine and students get used to offloading it to a machine? And at some point, I'm pretty certain that the machine will actually do better than what's up here.  I don't think that point is right now. But I just don't see our field -- and I'd be curious if you do -- prepared at all to grapple with that.

Shiv: Yes, I 100% agree with most, if not all of that.  One of the questions I've been grappling with, and I'm sure you are as well, is how do you teach this, right? How do you teach this when the compounding is occurring on the order of days, weeks and months, let alone years? A decade ago, when we were starting Osmosis, we were trying to get concepts like hidden curriculum, flipped classroom, multimedia learning, spaced repetition into the vernacular among faculty. Fortunately, because of what academic medical education has done -- but also private companies like Osmosis and many others -- it's pretty much well accepted. A lot of schools agree that medical education be more efficient. So, we're happy about that. But those still took years as opposed to now, which is like weeks and months. How do you make a course to teach faculty about AI and medical education, or get students to start experimenting with it in a way that doesn't get them kicked out?

Dr. Rodman: Yeah, that's a great question. I should say we still have sixty-minute lectures here at Beth Israel. So, it hasn't gone away, I can tell you. So, there is no guidebook, right? We're building the airplane while it's flying and I don't know that we have the luxury to wait and see, given that our students are using this right now. I can tell you what I'm doing and what our residents are doing.

Our residents are great and they've actually founded an interest group and now they have like four different tracks in it. They have a curriculum in artificial intelligence and in four different domains that goes across the year, and they've created an academic medicine half day for every single resident in our program so students can grapple with artificial intelligence and what it means for the future of their careers. At the same time, me and Zaheer Kanjee and Byron Crow have integrated LLMs into our clinical reasoning conferences. We actually do adult learning. We do like think pair shares informed by a lot of adult learning theory to get the residents to actually interact with it in a supervised setting and also a HIPAA compliant manner, and then have discussions about how they can use these technologies and what it means. We had one just a couple weeks ago.

So, that's what we're doing and like, again, do I know that it's the right thing to do? No. There's no guidebook for this. I'm very curious. I hope that @medtwitter sticks around because a lot of this is happening siloed at individual institutions and I hope that we all talk about it and over the next months to a couple years come up with a standard way of doing it. But things are changing so fast that what works in 2023 probably will not work in 2025.

Shiv: Yeah, totally. And oftentimes we borrow things in medical education that have been around in other tech areas like media tech or FinTech. A lot of creativity and innovation is just borrowing ideas from other fields and combining them, not coming up with original thoughts. One of the things that I've been grappling and thinking a lot about is again, what is the role of the faculty, the med student, the patient or the practicing clinician? You talk about @medtwitter...I'm wondering when the first AI-generated faculty member is going to be around contributing to the discussion or dialogue, but actually isn't a real person.

Dr. Rodman: Maybe they are already, and we just don't know.

Shiv: I mean, probably not because the money's not there as much, but there are AI influencers already that are raking in millions, if not tens of millions. That's raising the question of do we need humans if you have virtual avatars with great voice synthesis, and they can appear exactly how they want to appear so that everyone has their own personal tutor. Kind of what Sal Khan talked about with the Khanmigo.

Dr. Rodman: I was going to say theoretically with an AI agent, you're gonna have a personal tutor. This is getting to the exciting things. Like, if you talk about the way that we evaluate people in medical school, we know that it's not good. We have ways that are incredibly labor intensive, like the STEP exams and standardized testing. We know they're not good. It's just what we have. So LLMs likely will give us an ability to give more meaningful longitudinal evaluations. Of course, then you take it to the next step, and you're like, “Well, if it can evaluate and it can teach, can you not combine those in an agent and just have your personal tutor?” But, I'm not a tech person. I'm a historian and an educator. So, I have no idea the feasibility of that.

Shiv: I'm sure you like that quote, “history doesn't repeat itself, but it sure as hell rhymes.” I forgot who said that. 

Dr. Rodman: I think it's apocryphally put on too many people. No, it's true. What I truly think -- and I'm going to sound crazy on a podcast -- but the way that we currently conceive about both disease and medical education came at the beginning of the 19th century with this idea that disease lives in certain places in the body and we perform exams and do diagnostics and talk to people to figure out where that disease is and then that leads to treatment, yada, yada, yada. 

Then because of that, our education is focused on the scientific basis of these investigations, right? That's why we learn biochemistry. That's why I've had to learn the Krebs cycle so many different times. 

But we may be entering a period where now we collect so much data and we've created machines that are able to analyze this data in a way that's now separate from human understanding. We may be entering a new period of what the cognitive work of a doctor is. And again, that's an oversimplification because it's not like the cognitive work has been the same for 200 years. It has not. It has changed and it's changed in very real ways. But we might be entering that period.

Shiv: I recently started reading the book -- like half the world seems to be doing -- Peter Attia’s book Outlive: The Science and Art of Longevity. He’s a physician who's trained at Stanford and Hopkins, and he talks about this concept of “Medicine 3.0’ which is not even preventative medicine, but proactive medicine, where you don't wait for somebody to have a binary diagnosis if you're diabetic or not. Everyone has some sort of insulin resistance over time or everyone has some sort of atherosclerosis and it's much more continuous than binary.

Dr. Rodman: Yeah, exactly. It's a much more nuanced nosology that a human could never do, but a computer could do. If you look at how we talk about renal disease, we talk about CKD now. That's not a pathologic diagnosis. It's like a function diagnosis. I don't think this is a stretch of the imagination that twenty years from now, a lot of diseases will have been redefined or there may even be associations that there's not an etiologic relationship that humans understand. It's just there in the data and we can see that this relationship exists. Once that happens, that's a new nosology. But this is getting way beyond what we're talking about right now.

Shiv: But that's ultimately where it is going. I've always made the point that medical education is great, but what makes it different than other forms of education? I summarize in three ways. One is it's too vast for anyone to know. Somebody operating from first principles could understand a lot of physics and rederive that. Medicine is just too vast right now. Nobody can read all the papers. Number two, it's dynamic, right? Nobody knows what COVID-19 is, and so new diseases, new procedures, new drugs are always being developed or discovered. And number three is it's high stakes, right? If you forget how to conjugate a French verb, you may embarrass yourself in conversation, but you aren't going to hurt a patient's life or your career, and so those are the things that make medical education different. 

But ultimately, the purpose of medical education -- it's fun to learn about the body -- but ultimately the purpose is patient outcomes and what do you do when maybe AI can help disintermediate some of that or do it better? Whether it's the most thoughtful, patient-present AI psychiatrist that operates as an assistant or just independently via other medium to normal psychiatrists, or it's AI researchers that discover drug combinations that we would never otherwise think about. I think ultimately, like every discussion on AI and medical education it gets superseded by AI and medicine as well.

Dr. Rodman: Exactly, exactly. And what the professional identity of a physician will be. Doctors don't like to have that conversation.

Shiv: Or it gets misinterpreted quite a bit. I remember a decade ago, I was at FutureMed and Vinod Khosla was speaking. That's the conference where he said the very controversial thing at the time, which is in a decade, 80% of what doctors do will be replaced. He was a bit overly optimistic about that, but it also was widely misrepresented as 80% of doctors won't be needed. Obviously we keep changing what clinicians do. So, here's a personal question. As you know, I'm a third-year med student right now. Say you're my attending, I'm on rotation with you, and we sit down and you're giving me advice on residency, on my future. What type of advice would you give me, especially given that AI will be better at diagnosing than I could ever be?

Dr. Rodman: I don't know the future, right? I can only be informed. You and I are both like, it's 1770 and this guy named Bichat is about to blow everything up. So, what I would say is that just like Dr. Khosla said, it's not that doctors themselves are going to be replaced. It's that a lot of the cognitive work is going to be offloaded and different professions will be affected differently. I think it's so hard to have a crystal ball and look. So, I would say go into what you're interested in. Be curious. Be really curious about how you can engage with this future, and I think that probably there will continue to be roles for humans in the medium to long term. But I don't have an answer, right? I don't know.

Shiv: Should I match into radiology or not?

Dr. Rodman: Yes, but interventional radiology is always going to have a place in our system until robotics is like, yeah, do you want to do RADs?

Shiv: I don't want to do RADs, but RAD-ONC or interventional RADs are interesting. But yeah, anything procedural because that was one of the realizations, I think a lot of societies had over the past year since ChatGPT came out was that maybe a lot of what we do as knowledge workers will be replaced before traditionally blue-collar workers, right? Like the blue-collar trauma surgeons who may be listening to this or future trauma surgeons, I think are safe for quite a while.

Dr. Rodman: So, what I would say is that if you look at the cognitive work of most physicians right now, a relatively small amount of it is like these really tough diagnostic decisions. A lot of it is fairly humdrum...communicating, organizing, things that people still do pretty well right now, but honestly, we don't really train people for in medical school, right? We maybe talk about communication skills, but we don't really. So, I'm less worried about the job itself. I'm more worried about our professional identity, at least in my field in internal medicine. 

We pride ourselves so much on our ability to make diagnoses and a lot of our teaching is around that. I actually do think that in the reasonably near future, computers will be able to do that better than humans. I'll still have a job. I'm not worried about not having a job. But what does that mean for us as doctors and our self-conception when we're no longer doing that?

Shiv: One piece of advice you've given us twice now is just getting curious or staying curious. Where do you get your information from, if you can walk us through how you kind of scratch that curiosity itch? Are there books you're reading or have found very influential? Are there newsletters you're subscribed to? Basically, so our audience of students and faculty or professionals can keep apprised or abreast in similar ways that you are...

Dr. Rodman: Oh, it's so depressing. It's Twitter or X. I read a lot of preprints. It's finding a community of people and participating in that community. So, I talk to a lot of nerdy clinical reasoning people, a lot of nerdy diagnostic people, and we share papers back and forth and that's honestly where I'm exposed to most things rather than a specific book or a specific podcast.

Shiv: I'll put a plug in, since we're talking about clinical reasoning, to your research letter in June in JAMA the title of which is Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge, which I think is an important part of even this conversation. Can you give us a preview of some of the things your group is working on right now?

Dr. Rodman: Yeah, yeah. So, I need to be very explicitly clear -- especially if there are like AI experts listening -- I am not an AI expert. I'm a historian, and I study how people think. So, what we're doing is we're designing experiments that are informed by the way that humans think. And we're studying humans against the large language model. For example, giving them cases and sections and asking both humans and the large language model to solve it. Then, we're using a lot of methods that we've used to evaluate humans in a blinded fashion to see who does better. 

In the more exciting studies, were prospectively actually looking at how human cognition has changed when humans use an LLM versus the standard of care. For the near term, it's going to be used as decision support, right? It's going to be running on top of what humans are doing. There's some interesting behavioral economics data out here. I'm sure you saw the chest X-ray piece where even though the AI did better -- it performed by itself better than the radiologists in reading chest X-rays -- it made the radiologists perform worse. 

Then there's some interesting data in non-medical fields about how people with different training levels will have different increases or even decreases in their performance when using an LLM. I think there's a lot of interesting questions out there about what does it mean for a human to work with these tools, and that's what I'm focusing on right now.

Shiv: That’s really good nuance for sure. It has been a theme, ever since we made our first human-technology interactions. EMR is usually the big scape goat about how much worse they’ve made bedside manner, or you know, burnout or moral injury, or at least contributing to that. So it'll be interesting. Hopefully this is one of those things that reverses the course and helps clinicians connect better with their patients because they don't have to transcribe every note or be so focused on the computer.

Dr. Rodman: There’s a best-case scenario there which is like the Star Trek future, where we’re all on the Enterprise with Dr. “Bones” McCoy, and technology just works to enhance the human doctor relationship. There’s also a worst-case scenario when AI would be used to optimize finances of a hospital that just crowds out what’s left of the human experience. I think that’s what I’m so motivated about. Like a pharmakon, there really is a way these technologies could dramatically improve what it means to be a patient -- and hopefully what it means to be a physician -- but the same technologies could be used to make things worse. 

Shiv: I want to be respectful to your time, so I only have two more questions for now. The first is, zooming out of medicine, what are your thoughts about AI in general? Another historian I like to follow, and he has been very influential to all of us, is Yuval Harari. A lot of what he’s been warning the world is about AI, AGI, and what makes computers different from any technology, like nuclear weapons. What are your thoughts? I’m not asking you to prognosticate the future, but what do you think about AGI in this potential future beyond medicine?

Dr. Rodman: So, Shiv, I should ask you what you think. This is an area when I’m truly agnostic. You know on Twitter, or X, how everybody was talking about LK-99 and all the excitement about superconducting at room temperature. Well, I have no way to parse what I should think about that, and that’s how I feel about AGI. I don’t have a sense. I truly can't answer the question.

Shiv: I think that's great. I'm glad that you did not choose to because we're on Twitter, you know, having these conversations and everyone feels like they need to have an opinion on something and that's partly contributed to the world we live in today.

Dr. Rodman: I know a lot about how the human mind works. I know a reasonable amount about medical education. I don't know anything about AGI. I definitely don't know anything about room temperature superconductors.

Shiv: I feel like I know a lot more about both those things than I ever thought I would at this point.

Dr. Rodman: Yeah, I know more than I did a couple of weeks ago.

Shiv: But the other piece is because you and I are both obviously on social media -- I don't know if you find this -- but I think about how few of our colleagues or just the general public actually know or pay attention. It very much feels like an echo chamber in some ways. Do you feel this at all, or is it pretty much part of the zeitgeist that everyone on the BIDMC faculty is talking about AI at this point?

Dr. Rodman: No. For anyone who wants to shock their own faculty into waking up...in April we held a case conference -- and this is just like a few weeks after GPT 4.0 was released -- where we had GPT 4.0 actually as a discussant up there alongside with humans and we took questions, justified this diagnosis, interacted with the other discussants. I would have done it very differently now because I have a lot more experience, but the purpose and the reason we did that was to shock people out of complacency, to show the power of these tools. I mean, it's August now, so I think a lot more people have had some experience or have read about it and are realizing the impact. Honestly, ChatGPT is so easy to use, I don't know why more people aren't just experimenting with it. But I think people need this education and they need to wake up because these things really are powerful and they're going to change the way that we practice.

Shiv: Yeah, I agree. And I'm right there next to you, trying to get as many people educated about this stuff. My last question for you is, is there anything else you want our audience to know about you, about the work you're doing, about AI or medical history, whatever? Open mic for you.

Dr. Rodman: Oh, an open mic for me. Well, that's dangerous. I did not think about this ahead of time. I'll try to end on a positive note. If being a historian has taught me one thing, it's that being a physician as a profession has changed a lot, but there are certain fundamental things that are very similar, both cross-culturally and over a long period of history. A lot of that has to do with the relationship that physicians or healers have with their patients. So, being optimistic, AI may actually be a tool that we can use as physicians to enhance that really important relationship. 

The actual day-to-day work of a physician has changed dramatically, and we shouldn't necessarily be scared of change if that's what helps us take better care of our patients. I think that, especially for medical students coming up, it is time to be open-minded about what our professional identity is; what it means to be a physician. 

I do not think that we can have this 20th century view of the physician as the top of the ivory tower, looking down, giving pronouncements, being the expert in all things. We have to start reconceiving the physician as acting in a multidisciplinary fashion, including working with artificial intelligence, but also reaching outside our field. 

You talked about getting educational innovations from FinTech. Well, there's many other fields out there that I think we need to embrace and take the wisdom from. Remember, what are the things that make a physician the physician? It is that relationship with our patient. Have that as our North Star as things are dramatically changing around us.

Shiv: I love that. That's a hopeful note and an important note to end on. So, Adam, I'd like to thank you for taking the time to be with us on the Raise the Line podcast, and more importantly, for the work that you're doing to educate so many people about this important topic. I'm looking forward to reading your upcoming articles and sharing them widely at Osmosis and beyond.

Dr. Rodman: Well thanks, Shiv, and you guys are doing great work. Like I said, I read your article in Academic Medicine on Osmosis in global health education years ago. It was a big inspiration for what digital education can do.

Shiv: Well, thank you so much. We’ve definitely been the beneficiary of working with so many awesome, talented, passionate educators like yourself over these years. So, with that, I'd like to thank our audience for checking out today's show and remember to do your part to raise the line and strengthen our healthcare system. We're all in this together. Take care.