Are Micro-Assessments the Future of Testing? - Sebastian Vos, Chief Business Officer of Turnitin

RISHI DESAI: Hi, I'm Rishi Desai. Today on Raise the Line I'm happy to be joined by Sebastian Vos, currently the chief business officer of Turnitin, who was recently the CEO of ExamSoft. ExamSoft helps academic institutions improve student learning with secure assessment tools and software that provides valuable data insights and reports. His career has focused on identifying opportunities and implementing strategies that enable businesses to achieve their full potential. Thank you so much for being with us today.

SEBASTIAN VOS: Thanks for having me. It's a pleasure to be here.

RISHI DESAI: I'm curious, thinking about your role with ExamSoft and Turnitin, what insights have you seen in terms of how education has shifted or moved in the last 12 months that may be not in the mainstream discussion, things that are maybe nuances that you would pick up on that most people may have missed?

SEBASTIAN VOS: It's a great question. What I thought you were going to ask is how I've seen it shift, because I've been at it for a while, at EdTech. Look, the first thing that happened in selling EdTech largely was the content. So, content moved from whatever….print to digital format, and that was sort of the first transformation. Then, secondly, I would say it was a tool transformation. Thirdly, it's been I would say activating content within the tool.

So, we're sort of in this third phase, which is where we are today. I would say the biggest shift that we're seeing is that for a lot of programs, I think they’ve struggled to make the full-on decision top to bottom that they were going to join the 21st century and grab and really use digital technologies to improve the teaching/learning process.

What the pandemic has done is, it's broken that model. So, it's no longer, "Should we be using digital tools, or how are we going to use digital tools?" It's like, "Which tools are we going to use, and how are we going to implement them that makes the most sense to help our mission in educating students, but also not breaking the faculty?" Because they're under a huge amount of stress.  They were told two days before spring break, "Hey, when we come back from spring break, you're teaching online full-time." Many of them have either had no experience in what that looked like who had to go to a hybrid model and I think the ability of that transition is very difficult on the faculty. Not all faculty, of course -- some of them get it, no problem. But I think, especially in health, I think there's been a big shift for them.

RISHI DESAI: Are there any features that you've come to realize about the faculty that have been able to adapt well, versus those that have not been able to adapt well? Any sort of predictive features?

SEBASTIAN VOS: No. We are ExamSoft. We are an assessment product. So, the paper and pencil was largely our biggest competitor. People are used to blue books and bubble sheets. Historically, we almost had to do two sales. The first sale is, "You should not be using paper and pencils. You should be using technology." The second sale was, "You should be using ExamSoft's technology." COVID did away with that first discussion because they realized they couldn't assess their kids in an appropriate, fair way when they're remote. So what you're seeing is, that's gone away. It's just a matter of what tools they want to use and when, and how they were going to pay for it, because budgeting is always a problem.

To be fair, last spring…I call it the toilet paper buy. People just bought anything they could. Everything that wasn't nailed down got purchased. Now in the beginning of this fall, you started seeing schools being much more thoughtful about what they're purchasing. I think they're continuing that process of looking at what is actually making a difference, “What tools and content that I've purchased really are helping me?” You know, the story for me is novice to expert. How am I taking my students from novice to expert and what tools are facilitating that for me, or either helping the students along that journey or helping my faculty to help my students along on that journey?

You asked for features or signals. It's about who's willing to go through that change management? Because change is hard and you have faculty who have taught a certain way. They were using the same materials they've used for a while, and all of a sudden, none of that works because none of the kids are in front of them. They're all remote. How do you engage them when we're sitting at home and we're all talking and communicating via Zoom?

So, it's about how do you keep your students engaged and how do you know they're learning what they need to learn, and how can you assess them in a way so you know where are their strengths, where are their weaknesses, and are they progressing? If we're talking about health, are they ready for licensure or not? If I'm talking about the students, if I'm an institution, I'm like, "Am I doing what I need to do because of that accreditation, I'm not going to get a pass from my accreditors. So, how do I make sure I'm still on path to make sure that I'm going to clear when the accreditors come in and actually do their reviews?"

So, I think we talk about all of that. With us at ExamSoft, one of the big things we talk about with our clients is about the data. Are they willing to engage about having data discussions? If they are, then they're really ready to buy, because that's, at the end of the day, what digital gives you that paper can’t. Does that make sense?

RISHI DESAI: Yeah, it totally makes sense, and it kind of strikes me…there's an old cartoon that I've seen where a guy's looking for his keys underneath a light and they say, "Well, where did you lose your keys?" And he's like, "Over there in the dark area." "Why are you looking here?" He's like, "Well, this is where the light's at."

SEBASTIAN VOS: Right.

RISHI DESAI: And it's one of the things that strikes me, with exams, and you talk about going from novice to expert, I'm curious, do you get questions from faculty around things like, “How do we make sure that this exam, this digitally proctored exam, is capturing the things that we need to capture? How do we know that what you're counting counts, and that what counts can be counted?”

SEBASTIAN VOS: That's a good question. So, one of the key features of ExamSoft is everything can be tied to a taxonomy. So, if you're a medical program, you can tie every question you do to the USMLE. Or you can take it back to standards and each question can have an unlimited amount of categories. So, one set of categories can be your syllabus. One can be your course level objectives, and another one can be USMLE standards or the NCLEX standards or whatever standards you need. One can be Bloom's Taxonomy.

So, early on, you're just like, “Can you regurgitate the knowledge? Are you gaining the knowledge you need?” And then, you can work your way through Bloom's, and then you can keep working your way through to make it towards the end -- because you've tagged everything as you've gone through your assessment -- you can actually start seeing, “Are my students progressing and learning what they need to learn? Where are their strengths? Where are their weaknesses?” And you can share that with the student.

But as you look at your data, you can get feedback about are my questions any good? If everybody's getting all the questions right, probably your assessments aren't right. If you know everybody's picking A and B on a multiple-choice question, C and D as distractors are rubbish, so you probably want to go back and revisit that. There's a lot of tools we provide to get that data so you can expand the spotlight, as in that cartoon. We try to move that light and make it as broad as possible and then also allow you to focus in exactly where you want. Because we could say your keys are offline. Let's just focus my keys over there, because for Sebastian, that's his weakness, that's where he needs to focus. I'm fine on the knowledge on the cardiovascular system, but my ability to synthesize is terrible. So, that's where Sebastian needs to focus. He doesn't need to worry about anything else. And for Rishi, it's completely different. He needs to focus over here and he has a totally different set of study plans we need to put in front of him.

Then, as an administrator, I can go back and say, "God, every time we have a question or assignment that touches pharmacology, our average grades go down. I might have a curricular problem in pharmacology, so that’s something I can address." Hopefully that makes sense.

RISHI DESAI: It does. It does. You were very kind with the way that you framed Rishi as a student, because if you knew Rishi as a student, you'd realize that he needs a whole individualized education plan and...

SEBASTIAN VOS: Oh, look, yeah, I'm remedial as well. So, my spotlight is really broad.

RISHI DESAI: Right. It also kind of gets me thinking…you talked about the fact that you're obviously moving more into digital and using technology to assess students. What about the notion that "Well, when I met with Rishi in person, lovely kid, super book-smart, but gosh, he's just so awkward with patient interactions, and it doesn't really engender a lot of trust, and I can't get him to give eye contact for the life of me.” And especially with a nursing job or a medical job, those kinds of pieces are critical. How does one assess that effectively using technology? And I don't mean to say it has to all be one tool, but what are your thoughts on that broadly?

SEBASTIAN VOS: Well, it's super important, right? And especially if you're thinking about health or law or a lot of these careers where you have to interact with humans. It's just, you have to, right? You're not an actuary science person who's just going to sit and churn on your documents all day. Unless you're a radiologist where you're sitting in the dark, most medical professionals have to interact with humans. And so, that's why ExamSoft provides a tool that does that. It's called ExamSCORE. It's like a rubrics-based solution where you make whatever subject the object.

So, I can track you with categories. I can come back to you and say, "Here you are in your patient side, your skill side is phenomenal. Everything you do on the skill set is fine. Your human side is terrible. Here are your marks, and by the way, you're not getting any better because I've got the data from day one, when you didn't know anything, to day 365 one year from now, and you're failing in the same kinds of ways." And you can track that. Or, "You're getting better. You're making eye contact, but you're not talking to the patient."

Part of that tool is also -- we call it an auto-fail function -- where say you're about to administer a drug and you haven't checked the arm band. I can auto-fail you. You can continue with the skill. So, you did all the skills right, but you still failed because you didn't check the arm band. Or you didn't wash your hands before you started, or you didn't greet the patient.

RISHI DESAI: That makes sense.

SEBASTIAN VOS: It's about assessing the whole experience. And that depends…program by program it's all slightly different. Whether you're a phlebotomist or whether you're a doctor, there are differences in what you're trying to do and the whole learning process, and we try to provide tools to assess that whole experience.

RISHI DESAI: I went through medical school two decades ago and when I was going through, the technology in my phone was not what it is today. I had a pretty basic phone at the time and its main job was to make calls. Today, its main job is to get me data. So, nowadays, I can't even imagine seeing a patient and not at least referencing some data source at least once. So, it seems like more and more, it's not about what's between my ears, it's really about, “Can I access the right data source to get what I need at a given point in time?”

So then those soft skills end up being the critical ones. Talk to anyone nowadays. I think the biggest problem is, the doctor's hardly visible because they're behind a computer screen because they're referencing something in their EMR. The nurses generally get better scores on this to be honest, but still, quite busy with other stuff and maybe inputting data into a computer screen. So, that social piece seems like it's more and more and more relevant. I'm curious, do you get that signal from schools? Are they asking and inquiring more and more about soft skills training and soft skills assessments now versus let's say a decade ago?

SEBASTIAN VOS: Yeah. It's a good question. And I don't hear a ton of that. I think some more schools are doing OSCEs (Objective Structured Clinical Examinations), and on the medical side, I would say in Europe, they've been more driven towards OSCEs for longer. I think you're starting to see that much more, whether it's patient actors or simulators or doing something where you have to interact with humans, because of the phone and today’s generation -- as soon as those words come out of mouth, I feel old and stupid -- but they're more interacting with their phones and looking down.

My daughter…if I go out and she's with seven of her friends having a good old time most of the time, they're on their phones. They might be chatting with each other on their phone, but they're all on their phones looking down. When I was a kid and went out with seven friends, it was just loud because everybody's talking and yelling at each other and doing whatever. I think it's that generation of, “You have to interact with other humans directly” and giving them that skill to do so is becoming more and more important, though I haven't heard it as much outside of maybe, as I said, sideways coming at it through using OSCEs or more simulated or with either actors or the mannequins, those kinds of activities, and getting them out of the textbooks.

RISHI DESAI: What do you imagine would be the key needs, then, of ExamSoft or Turnitin in the next five or 10 years? What's the direction these platforms are going to go in to help meet the needs of folks like your daughter?

SEBASTIAN VOS: I think you're going to still have the big set pieces, as they were, those final exams. The thing that we're working towards is low friction assessments. You're on your rounds, you're going to walk in and you don't know what you're going to see. Let’s say it’s a first visit post-op hip surgery. I've never done this before. What do I do? You hand your phone to your clinical advisor, and you start in. It might be that all I'm going to get graded on is my communication. How did I handle the patient? How did I talk about it? Not how well you actually did the medical aspect of it. But that could also be part of it. And then gathering that data and seeing how you progress from whatever…your first year to your third year, fourth year, fifth year. How are you scoring?

We should see you improve. It should be pretty rubbish in the beginning and get better. It shouldn't be like, “Well for a first year he did pretty good.” No, that's not good enough. We should be marking you as a doctor. You're here and you're not good at it. You need more experience. We need to train you on it because we would have that same expectation of you on the medical side of it. Do you know what you need to do? Have you done that correctly? So, it's that very quick, frictionless three or four minutes on my phone. Here it is. “Oh, I don't even know you. You're going to take my phone. You're going to do my assessment. I'm going to do this thing.”

I'm going to have 100 of those over my career as a student, as well as those big set pieces that are also in there. And that's not just for medicine. You can be a car mechanic, and it's like, “Am I turning my wrench correctly?” And it's very quick. We just call it frictionless. It doesn't take anything to get started and work on. I don't want to call it authentic assessment. I don't quite think we're quite there, but it's that very quick, very pointed experience.

Then, you go sit with your advisor and you've got 30 of these that you've done. And he's like, "Wow. It looks like your skills are pretty good. I see the improvement. You're not exactly where you need to be, but you're making progress and here's what you need to continue to work on so you can move on as a whatever medical professional. Or, “You know what, maybe you're just not good with people. Maybe you should be on the lab side."

RISHI DESAI: You know, I've seen a similar trend with learning. If you divide the world into learning and assessment, on the learning side, it used to be these large four-hour long and then one-hour long lectures and now they're these little micro videos. And with the assessments, same thing. I'm in pediatrics and now I do my quick quarterly assessment instead of a five-year assessment. Do you see that happening more in the U.S., or do you see that happening around the world? For example, are we at the bleeding edge of these frictionless assessments here in the U.S., or is Canada and the UK and Australia way ahead of us on that front, and we're just kind of playing catch up? What's your sense on that?

SEBASTIAN VOS: My sense is that I always think Australia is pretty advanced in some of the thinking they have. It's a small community. They're very global in their viewpoint. Because they travel, they'll do their residencies or postdocs in other places, and they seem to come back and take the best of what everybody's doing around the world and can get things implemented. You know, in Australia, we see a lot more of this drive to do authentic assessments, so much less of “I'm going to fill out a bubble sheet,” and much more like, “We're going to have a conversation and it's going to be almost like a super OSCE.”

But instead of maybe a carousel, I'm spending an hour doing one thing or half an hour doing one thing. In the UK in some of their institutions we see them on this micro assessment. They seem to be taking a lead. The University of Liverpool is doing a lot of interesting work there. I'm sure it's happening here in the U.S. I don't think there's a groundswell for it yet. Maybe because of COVID we will be hearing more about it. I just think there's so many programs now that are struggling to figure out, what do we do? And with COVID, I think normal undergrad learning is going to move more towards a micro assessment. There'll still be the big finals, but instead of two big exams and five papers, I think they would much rather have a series of lightweight assessments that go on and still have a final.

RISHI DESAI: Yeah, I completely agree with you. I can definitely see it moving in that direction. I believe it might even get to the point where it's just a string of micro assessments all the way through that may not even end on the last day of class but may continue on at a lesser frequency beyond that last day of class. Almost like kind of a CME applied to undergrad.

SEBASTIAN VOS: Yeah. Well, I think one of the complaints hospitals always have when they bring on people out of the universities is how expensive it is to have all the training they still need to do.

RISHI DESAI: Exactly.

SEBASTIAN VOS: Right? And you've lived it. I've only seen it. But if you go talk to the chief nursing officer at a major hospital their number one complaint is, “I bring in all these nurses from great schools that have smashed their NCLEX, they're super smart, and I’ve still got to train them for three months before they're actually useful.” I think that's where you're going to see that drive towards micro assessments, that they're going to keep pushing that on. It may not be necessary for licensure, which I think some people would see.

It's going to be almost like a CME or CNE or some sort of CPD kind of aspect for you to keep progressing. To keep my magnet status as a hospital, I am going to need to continuously assess my staff all the time. I wouldn't be surprised if we started seeing it at the big hospitals and big programs. Brigham and Women's will do it before you start seeing it at the community hospitals. But that's what I would expect to happen. You're right. Even though it stops in the classroom, it keeps going on.

RISHI DESAI: Yeah, and I think it's for the point you mentioned. I mean, employers want to try to trim down that three-month training window to 10 weeks, eight weeks, six weeks. They want to see any incremental improvement there that they can. So anything that proves it can trim it down, I think will be worth it to the employer, and that's what will get financed. Then, once there's a business engine around it, then you see more education going in that direction.

You know, I do want to give you time to teach us something. You're aware we're a teaching company, and one of the things that I like to do is to fill in any knowledge gap that our learners may have, that I might have, with anything that may be a myth. It could be something that you commonly have to teach or remind people of. Is there anything out there that you feel like you could impart upon us that would help fill that kind of a knowledge gap? And it could be on any topic.

SEBASTIAN VOS: Maybe I'm too humble. I don't know what I know. Look, I would say, I think the teaching/learning process is amazing. The problem we have is -- it's a joke I tell -- but if you could go in your Bill and Ted's Excellent Adventure time machine and bring Socrates and drop him in a classroom, he’d know exactly what's going on because education hasn't changed in 2,500 years. And that's okay. The model seems to work all right. I'm not purporting “sage on the stage” versus “God on the side.” That's too trite. But I think education is lagging behind what is happening in the rest of the economy, which is data-driven decisions. I think anything we can do to use good outcome data to help improve education is super important.

An example I'll give you is, we have a client that took our assessment data over time and merged it with their Student Information System data to get all the demographics and whatnot, and then regressed it so they could identify students that were going to be retention and attrition risks. They identified what those profiles looked like and then made interventions in August instead of waiting for January or December when they're going to fail out. They used that data to help drive those learning outcomes.

So, how can you help your students? How can you help your faculty if you're in administration? How can you help the whole school become a better institution? Because at the end of the day, if you're not giving value for money…if I can't get students who can become employed, whether you're in health or whatever you want to do, if my faculty aren't engaged and wanting to help that student on those journeys and be on that same kind of journey, how can we help those things?

To me, that's all about the data. You have to have good data, right? It's not about rubbishy data. So, for me, it's about getting at the data. Scratch it at, work at it, figure out what makes the most sense. Our tool at ExamSoft gives amazing data on helping those kinds of things. But there are other sources of things that work. Bring it together, think about it, because every other major part of the economy, that's exactly where they're going. Amazon knows more about you and your shopping habit than anybody else. Educators can have that kind of knowledge without being creepy about it, but can share that with you so you can go on your personal journey.

So, if you're in medical school, they know exactly where your strengths and weaknesses are, and tell you, "Don't bother studying that stuff. You know this stuff. You're smashing it. Where you're having problems are in these areas, at this level at Bloom's Taxonomy. This is where you need to spend your time." If we can get education at that point, I think our education system will have taken a giant step forward. And quite honestly, we, as a U.S. education system, would probably be way ahead of almost everybody at that point, because nobody -- outside of some of the smaller systems – nobody is really using data the right way holistically.

RISHI DESAI: I couldn't agree with you more. And I'm going to ask you for a specific number here. So, be forewarned.

SEBASTIAN VOS: Uh-oh.

RISHI DESAI: Let's say 1 is sorely disappointed, and 10 is just thrilled. How do you think Socrates would react if you dropped him in like you just suggested? Do you think he'd be kind of enamored with our current educational system, a full 10? Or a 1, because gosh, what are we doing given all the other technology that we have? What is your sense?

SEBASTIAN VOS: You know, I think it's almost unfair because I now have to average out everything. I think there's some programs that are doing amazing things, and if I took the average program, I think he would say, given everything that we know, we are doing an amazing job using an old technology to get it into kids' heads. So, in that sense, probably a six or a seven, or a five or a six. It wouldn't take much real effort to get that to a nine or a 10, but I think we just have to use better tools. There's so much information. And as you were saying about how long ago you trained…I'm old. When I went to school, we just walked around and didn't know stuff because Google didn't exist. I tell my daughter, "Hey, when we were a kid, we just didn't know stuff, and you just would walk around and you would find something out when somebody told you, like, 'Oh, that's amazing.' Because if it wasn't in Encyclopedia Britannica, you had no way of finding anything else out.” Now you have Google. You just Google everything.

What you need to have is better tools to determine is this resource right or wrong? Is it reputable or not reputable? But at least you can always find an answer to something. So, we need to move on beyond the aspect of do you know the answer? You don't need to know the answer. Google can tell you the answer. You can know the CDC is a good site. If they have something on it, that's a reputable site. I can move forward. Or NIH, or a good journal site. What I need to figure out is, based on what the knowledge that site is giving me and the person in front of me or the situation in front of me, can I synthesize it and then apply it? Those are the skills that I'm not sure we're doing a good enough job on.

RISHI DESAI: Yeah. That's a fantastic point to end on. And you did mention the phrase “rubbishy data” earlier, and it kind of gets to the idea of distinguishing rubbish from value as the critical skill.

SEBASTIAN VOS: Correct. And I think content is the same way. So, how do you know you have good questions? Just because everybody scores 100 doesn't mean they're ready to take your board exam. I can’t tell you how many clients are coming to us and say, "I don't know why my pass rates from my boards are so poor. My students are doing great." Well, let's look at your write-ups. Everybody is doing great, of course, because your items are all knowledge-based and you're basically teaching to the test. And so, of course they're going to do well in your class. Then they sit for the board exam, and they get smashed because they're not ready for the exam.

RISHI DESAI: And in many cases, probably not ready for life after the exam as well.

SEBASTIAN VOS: Correct.

RISHI DESAI: Well, listen, this has been an absolute pleasure chatting with you. It's been a lot of fun for me. Thanks for sharing your wisdom and your ideas. I think that they're very powerful ones.

SEBASTIAN VOS: Well, Rishi, thank you very much for the invitation. I really enjoyed it as well.

RISHI DESAI: I'm Rishi Desai. Thanks for checking out today's show. Remember to do your part to flatten the curve and raise the line. We're all in this together.