Stanford Healthcare taps AI to cut burnout, boost efficiency in patient responses

Hi, welcome to CIO leadership Live. My name is Lucas Mearian. I'm a senior writer with Computerworld magazine, and I'm at the CIO 100 symposium and award show in Scottsdale, Arizona. I have with me one of the award winners, Aditya Bhasin.

He's the Vice President of Software Development at Stanford healthcare in the Bay Area, they run about 300 facilities, including two full service hospitals, and they rolled out an AI solution that automates draft responses to patient inquiries about billing.

And most organizations are only piloting AI right now, and considering it's healthcare that's one most highly regulated industries of any vertical, hands down. So I've got to hear more about this.

Tell me a little bit about the project, how it started, what were some of the challenges and what were some of the results, the benefits?

Yeah, so I think, you know, Stanford Medicine is a really unique place, I think we've got three missions. There is education, research and clinical care, and AI is transforming each and every aspect of those particular mission.

You specifically asked about what we did for billing, you know, and our journey with AI started about, I would say, a couple of quarters after chatgpt was announced, and it was basically the organization decided to double down across the three missions, using that technology to transform the core business, one of the first initiatives which we undertook in that particular as part of this program was to help our clinicians with their burnout.

I think post covid our portal engagements and a patient's choice of using the digital channel just increased exponentially. So one of the first things which started to happen was we had physicians who were getting overwhelmed with all these messages coming in from our patients, right?

So when this technology came about, you know, the ability to use large language models, one of the first things we rolled out was, how do we help our physicians?

And we started creating draft responses, and that, as you mentioned, in a regulated industry, required a lot of soul searching and thought process to go in. But essentially, whenever patient messages our clinicians, we create a draft response which kind of helps the physicians.

You know, reduce the physicians cognitive load right as they're looking at, you know, how they should be, because millions of these messages started coming in post covid, as people shifted to the digital channel. And that was super successful, you know.

And we can talk more about that, you know, later.

But as we are looking at the success of that for the clinical staff nurses, Ma's, you know, and then our physicians, we also started noticing that there are unique opportunities with, you know, once people get their care, one of the other friction points, and when you talk about patient experience, is billing.

You know,

we all have experience in healthcare. Woof, that's right, yeah. So, you

know, I think we've all experienced the complexity, and it's uniquely challenge, challenging for patients, and especially when you have proxies, when you're dealing with your own kids, or you've got, you know, one individual is a guarantor, but the spouse is on on your insurance plan.

The complexity of this, and as you have, you know, high deductible plans, the responses are billing reps have to give, requires a lot of understanding of the insurance, your unique insurance, what you have paid, and all of that was a lot of work.

So we said, you know, how do you apply this new capability to that, you know, niche area? So I think we were the first ones to attempt at that.

And we started off by actually doing what we, you know, classically talk about is, you know, gamba rounding, you go and you actually observe what people are doing.

One of the things we observed is, you know, given all the nuances which are there related to billing, is that the organization had created 25 different templates for the reps, right?

You know, if it is this step of a question, you roughly start with a template, you obviously enhance from that particular template, and then you create a response for them.

Should also realize that you know that that adds a lot of load to, you know, the work which is involved for the billing reps and.

So we took all of that into consideration and basically automated that entire process, wherein a draft response is now created, which takes into consideration your insurance, your specific payment plans, where you are in your deductibles, and then chooses amongst these, out of these 25 templates, the right template to curate a response.

Does this also allow physicians to adjudicate some of the payers? I understand, payers are also using automation to deny claims, and physicians will go to bat for those patients. Does this allow you to do that as well? Yes,

so those are that's a great question. And those are further opportunities we are actually evaluating. This is right now.

We took all of you know, we started this as a pilot for our billing reps, so that when patients ask them questions about where they are, you know, in their payment plans, and you know what's going on with billing, this is targeted specifically for that particular use case, and we saw a substantial amount of success.

We started off with a pilot like we do in most of our rollout of our use cases, with about 10 billing reps and just over a span of them using this technology to respond to 1000 messages, we saw a savings of 17 hours. Wow.

And then we actually go out and we queried, you know, follow through with our patient advisory forum to kind of see that type of responses they were getting and the timeliness of those responses.

So the patients, from a patient experience, were really excited that they got quicker, faster responses. The billing reps were super excited about actually getting, you know, technology to help them curate a response and answer it correctly. And the organization came out ahead in actually time savings.

So we took that particular period, we ran this for a quarter, and then we basically rolled it out across the organization. So at this particular point, all of our billing reps have access to this technology.

I mean, most organizations are only piloting AI right now. They don't trust it enough. How accurate was this? It seems you always have to have a human in the loop. Somehow, what are you finding in terms of accuracy?

Yeah, so I think varies from case to case. And you know, the level of complexity on this, we found really high utilization, so about 60 per person of the use cases. Yep. You know, the billing reps have been absolutely excited about using it as much.

But as with every technology you know, there's a care, there's a maturity curve and how complex this situation gets, and how much do we want to evolve.

So as part of our standard processes, we also are continuously monitoring and further enhancing and tuning, you know, our prompts to kind of further tune. And, you know, get this

because this was an internal rollout. This isn't a cloud service, correct? That's right, yeah. So we

leverage, you know, at Stanford Medicine, we have access to all the large language models, and we obviously do it in a secure way, right?

So these are all HIPAA compliant environments, and we use the standard technologies, but then we are basically enhancing it for further by prompt engineering it for our specific use cases.

Now you had another AI project that you piloted and then rolled out, and that had to do with drafting test results for increase from patients that too. I'm assuming that the physician oversees those as well to make sure they're accurate. Tell me a little bit about that project.

What were some of the challenges and what were the benefits? Yeah, so

that was you're correct in mentioning that it's again, who are one of those innovative users of AI to kind of help physician burnout as well as improve patient experience. Now you can imagine, how many at an organization like Stanford healthcare, how many test results are ordered?

These are in the millions. Oh, yeah, right, annually.

When you have a large patient population and you've got 1000s of physicians ordering labs, images is standard part of the workflow for all healthcare today, and I think most of us can relate to the fact that when you know lab results are released to you as patients, and now as part of you know our legal requirements, we immediately release these lab results to the patient, so you get the lab results in your portal.

But if you ever. Ever try to decipher a radiology result, or ever try to decipher a complex blood panel, understanding it adds they come with a level of complexity, so you're waiting for your physician to really decipher that and give you some insight.

Now, when it's the same challenge, when you have millions of these coming in, and, you know, as these labs get more and more complex, it adds a lot of workload on your on your physician.

So if you've got family practice or you've got a specialist, but you've got up, you order a bunch of these labs to, you know, decide what you want to do next, right?

But then you have to read all of these labs and then respond to the patients, you know, telling them what you think is is going on.

So we took exactly, you know, we use the technology we use for creating draft results, and basically completely started understanding, you know, how can we help them create responses which would help their patients?

So at this particular point, the moment a lab result comes in, we also create a draft result for our physicians. Now, these could be complex Labs, which could just be around, you know, blood panels, they could be radiology related.

This is across numerous systems, so you have to connect them together in order for this to work right. You're

absolutely right, or pathology. So a lot of these complex systems all pumping information into our EHR, and then we using, you know, this homegrown framework, which we have created, to create responses for our physicians, and that has been super successful.

You know, in terms of adoption by the physician, like you mentioned, there's always a human in the loop, and we never force our physicians to use Right, right? So it is the comfort level, the complexity end of the day.

You know, there's a level of trust we have to build with the physicians and up. You know, the sacrosanct relationship is between the physician and the patient, and so the physicians on the hook. So we are always trying to basically help them, help their patients

between those two projects. What was the greatest challenge? I mean, you've got to be concerned, highly concerned with privacy and security. HIPAA, just regulations in general. But was that the greatest challenge, or was it something else?

So I think for most technologists, technology is the exciting part. Yeah, and you know, we want to apply every time we get a new tool, we want to apply this.

But to the points you may mention, you know, trust being in a highly regulated environment, how do we go about actually making sure that we are not exposing our organization to risks and simultaneously maintaining, you know, the right level of efficacy between the physicians and our patients.

So there is governance structures which are in place. And, you know, with the advent of AI and the utilization. We've got governance bodies right at the sea level, so all clinical stuff goes through that.

We also have, you know, a process by virtue of which we validate all of the, you know, the solutions we come up. We call it the form process, which is fair, useful, reliable, AI models, because with AI, we also see the aspect of one are our interventions fair?

Because, you know, you want them to be fair across all genders, all demographics, you know, name it, and you want to make sure that your solutions are equitable across all aspects.

And we know AI can go off the rails sometimes that's right, right?

So as part of this process, we also leverage ethics, bodies within Stanford mere medicine, to look at the responses which are coming and their way. And yeah, huge aspect of it is, is there utility to this?

You know, it's, you know, rolling out technology for technology sake, but what is the benefit? So, you know, all, all these solutions go into pilot stages. For example, even for our lab results, started with, you know, a cohort of our 10 physician informaticists.

Then we took it from there to, you know, a cohort of 24 these were the results were measured scientifically over a span of two quarters.

Then we took it to just our primary care physicians, and that was run for another three quarters, and only after that, we went to our specialty areas, and then we ran with five specialties. But.

Four, we decided to go enterprise wide, and at each stage, you know, the results and the feedback are measured when we get feedback that is built in to optimize, you know, the utilization of this tool, and a huge aspect of it is training, right?

That's, that's another one thing I was going to mention is this is something that impacts every vertical industry, and that is bringing the workforce along with the fast pace of change that AI is bringing that's right to an organization. How did you do that?

Thought when you know, I think the organization's been very forward looking in embracing AI, one of the things we did this is, a couple of years back itself, was making a secure version of AI, you know, we call it secure GPT, available to pretty much the entire organization, okay, right?

It had to be secure because we don't want people to do it. I was gonna say, right?

We But simultaneously, once the genie is out of the bottle, we also know that people would, if he didn't provide this, would be using commercially available tools to try their hands at it.

So we basically wanted to append that, and we just said we will make a secure version available. It's been hugely successful. Yeah, people use it for everything from administrative projects to clinical projects.

And that provided us a learning and, you know, the organization feels comfortable and feels supported in having this tool universally available. Then we worked on training we've created, you know, you would imagine at Stanford Medicine we train our next generation physician scientists.

And, you know, a huge part of AI is going into the next generation of, you know, of curriculum of how do we train? How do we educate? How do we create interactive sessions for our medical students and our basics, yeah, because you're

academia as well as your physician right there. And so we

took aspects and learning from that to actually create mandatory training for everybody in the technology group. How many people we talking about here? I talked talking close to 1000 people. Okay? And we took that training and then we basically rolled it out to everybody in the organization.

Now you're talking 10s of 1000s of people. This includes everybody from our facilities to our, you know, our operational staff to nurses and physicians. So that training is essentially available to everyone.

Because, like you mentioned, the rate of change of you know, technology is so fast, the biggest challenge for most people at this particular point is, how do you bring your organizations along?

You know, how do you roll out these transformations in the standard workflows, and how do you get your business leaders to simultaneously reimagine how work can be done, right? When you have these powerful

tools, this really disrupts workflows. It does right?

And so as much as we are trying to keep this interwoven into the existing workflows, how do we tell people that these capabilities are available? And these are, you know, early rollouts, which are enterprise wide.

And you know, since results have been positive, it's creating this feedback effect wherein we're taking on more and more, you know, projects to kind of transform. I

don't want to go down a rabbit hole with this, but I got to ask, because part of that training has got to be kind of reeling in the cost, because as input output, you're charged for those input outputs to the large language model system.

How did you deal with that in training? Because sometimes people ask question which then generates a follow on question, and then you have multiple people asking the same question around the organization, and you're getting charged every time they do that. How did you deal with that? Right?

So our version, which is available, we actually do not try and limit that particular aspect, right? Because part of it is we want people to get exposure, because we truly believe that once people get comfortable and understand how to use this technology.

You know that there's a lot of talk about a hallucination, but you're seeing we also made a bet that the rate of change would be so, you know, drastic, that soon things we were worrying about a year ago would not be concerns to the extent they are.

Are hallucinations happening today?

Also, absolutely, but you've seen just the models which have come out how better they've gotten over the last two or three years, and so if you make a bet along those lines, you want to create exposure for your employees to kind of start experiencing and understanding what the limits are there.

As simple as you trying to curate your email and. But you're forced to read that email and understand, you know, where it drifted off completely and where it is still sticking to the subject matter.

And so giving people that exposure, as much as, yes, people may use it, and multiple people may use it, but all of those things which you've taught about, how do you want to reduce when you kind of turn it into a solution and you roll it out, that's where we're bringing in the efficacies.

And we have actually seen, versus commercial solution, a substantial lower price points using the things which we have

to You're a software design and development guy, so I've got to ask this question, software assisted development, or AI assisted software development, Jive Jive development. Have you done it? How accurate is it?

Is your team learning to use it as well, and what kind of efficiencies you're getting it from?

Yeah, great question. And, you know, we get asked, Are we disrupting other people's you know, work by rolling out these solutions, right?

And my answer always is, we're not only, you know, transforming business in in the operational aspects or the education or the research aspect, we're also disrupting and transforming how we do software development.

So we're to the point you were asking absolutely yes, you know, I think about a year and a half ago itself, we made it available to pretty much everybody on the team.

You know, name the popular tools the teams are using those capabilities, and we do see benefits, right? I think again on the maturity curve where we can use these capabilities. I think again, if you play with this stuff, you learn where you can use it effectively.

And you can also see the trend lines of where you think you could use them in a few years, or, you know, in some cases, even a quarter or two later. And so we are using it across the care, you know, across all of software development.

And we definitely see advantages. Sometimes advantages are more in taking away the grunt work a software engineer has to do, yeah, that's just hugely impactful. There's just like we talked about the joy of practice of physicians.

You know, when we roll out these tools, we want them to spend more time with the patients, rather than, you know, spending time documenting stuff, which is kind of what we've done when we rolled out ambient listening.

So now a physician just talks to a patient and you know, the documentation automatically happens for them. Similarly, aspects of rolling out these capabilities for software engineers also brings back the joy of doing what we got into the business of doing in the first place.

You know, just writing a routine and having it automatically create a lot of test cases around it, that's a huge, you know, impact, right?

Yeah, we as software engineers were always able to imagine what the system would do, but we also knew it would take a certain amount of time. If we had the right processes, we were making sure it was secure.

We were doing all the right software design life cycle, you know, steps, yeah, yeah. Can it do all of it in a really complex ecosystem? No, but can it help in individual steps, and, you know, help speed those individual things up Absolutely.

And we are seeing all of that.

Would you say it's enhancing creativity, it's not eliminating jobs, as some have thought we have.

And I think repeatedly, you know, I think that's the fear across the spectrum of all the jobs. You know, physicians say the same thing. You know, you have concerns across back end jobs and software engineers have exactly the same kind of concerns.

I think it is bringing a little bit of joy back into what you do, and it's helping you get to things at a faster rate. And I think overall, the art of the possible is expanding. That's how I look at it.

Last question, sure, if there's one piece of advice you can offer an organization that's rolling out, AI possibly going from a pilot to live implementation as you did. What would that be?

I think having the right governance in place, making sure the right amount of training and change management aspects are taken care into I think it's a combination of having all these things in place. Otherwise, I. All the discussion points we've had. It creates fear. It creates anxiety.

How do you overcome that? And how do you, basically, you know, help the organization and get to the place they need to get

Aditya, thank you so much for your time today. I really do appreciate it, and it's great, especially having somebody whose boots are on the ground. You've you've done this so you really have depth of knowledge of how this is deployed. Thank you so much. Well. Thank you.

It was lovely chatting with you.