THE REAL TIME HEALTHCARE PODCAST · Feb 4 · 10:20 min

#28 - Andy Day, Real Time Forecasting with Machine Learning

In this Episode:

Andy Day and Jeff Terry provide examples for when AI is used in Forecasting. They cover examples including forecast census / staffing differential at a unit level, bed placement decisions, PACU arrivals / post-op bay placement and disposition / day of discharge. Andy and Jeff talk about the system of models that are used and re-run with real-time data to enable easier decisions for caregivers.

Jeff Terry
Hello and welcome. I'm Jeff Terry. Very happy to be joined today by Andy Day who's the chief technology officer of the command center team. Good morning, Andy.

Andy Day
Morning.

Jeff Terry
We're going to talk today about forecasting in real time using machine learning. Of course, we do that in Tiles for quite a few things to forecast census, staffing differential, PACU arrivals. It's very difficult to do in real time. We'll try to help you see what that means and how that's useful. So, Andy, to start us off, could you share a bit more about the types of things that we're forecasting?

Andy Day
Sure. Picture's worth a thousand words, I'll share my screen here so you can follow along. So a few examples of where we use AI and machine learning. We've been doing it for the last 10 plus years in healthcare. That forecast is used in analytic Tiles to help our users actually consume it in a meaningful way. So some examples, top left corner here, you see the census and staffing forecast Tile, where we're forecasting census at a unit level, look every hour going out for the next 48, as well as looking seven to 14 days out, so you can better set your staffing and adjust for situations that a forecast where your unit is going to be running out of beds or in trouble before it happens.

Andy Day
Top right, based upon all the patients competing for capacity within the system and your historical behavior over the last couple of years, who's most likely to win each of those contests. And where were they most likely actually get placed so that you can better adapt for what's likely to happen and to make more consistent decisions in bed placement. Bottom right, this is for a surgical daycare unit, which does both prep and stage two recovery for a large set of ORs and procedural suites. And we're forecasting both the supply and demand of both the bays and staff across the top, as well as forecasting at a patient level when a case might be disrupted in the future based upon the dynamics of the system and how they're playing out, before it happens so we can act upon it.

Andy Day
And then in the bottom left, for every patient that is in the house, forecasting what their most likely day of discharge is going to be, and their post-discharge disposition, are they going to go to a SNIF or an LTAC or go home with home health, et cetera. And so, these are just some good examples that we've done many more. As you mentioned in your introduction, we forecast OR holds before they happen so you can try to mitigate those situations. We forecast the supply and demand and wait times for support services in a lot of different places in the hospital, and a number of others

Jeff Terry
That's good. So just recapping, so upper left, we have census, forecasts and staffing differential. Upper right, we're forecasting the bed the patient is likely to be assigned. Lower right, we're predicting PACU arrivals and recommending post-op bay placement. And then lower left, we're forecasting disposition and day of discharge. Those are great examples.

Jeff Terry
Andy, something you mentioned I want to draw out. So unlike retrospective use of machine learning, where all the data is available, diagnosis, the lab results are all back, everything's there, when we're forecasting in real time, we don't necessarily have all the data, so we have to build the models based on what information we're likely to have in the moment. Can you say more about that?

Andy Day
Sure. Yeah. When you're doing machine learning for research purposes, or just typical retrospective data type approach, there's a lot of discussion around these things they call data lakes, right? Which is where you try to get as many data sources as possible, as large those data sets that you can get, with as many variables as you can identify, and focus on basically establishing common ontologies, curating the data as it's arriving, and trying to create basically an environment in which your data science team, your machine learning specialists, can go fishing. And what I mean by that is for a topic of interest, can we find an algorithm which is helpful?

Andy Day
And so, this works well for when you're doing sort of, like you said, retrospective historical data mining and large data sets. But when you're trying to do real time forecasting, this methodology really doesn't work because you've got a patient in the ED, they're being worked up, they have some data in, they're missing a lot of data that hasn't come back yet, and there's a lot of other things you won't know that you will know post discharge when everything's settled. So this methodology really doesn't work for real time because it's all about data latency.

Jeff Terry
Got it. And so, we've figured out, you and I were joking earlier, it's almost like instead of a data lake, we figured out what tributaries feed the lake and where should we fish? What's the best place to fish on the tributary so we get the information that we need when we need it?

Andy Day
Yeah. I'd even say we designed the tributary and build it to just [inaudible 00:04:53] because any more water will slow things down.

Jeff Terry
That's great. So if you would, take one example, maybe census forecast, which I think is what we do more than, or have done the longest. How do we do it? How does it work?

Andy Day
Sure. So when census forecast, for example, we happen to use machine learning technology called Spark, Apache Spark. And we use random [inaudible 00:05:18] amplification because we found that method to be the most helpful, but we also use other methods embedded. And one thing we found to be really important, we do a lot of that, that's called ensemble modeling where you combine multiple models in one model, and we do a lot of that. We also have found that it's important to model every unit at the unit level, having one model that runs for everything, spreads the mayonnaise, doesn't reflect the nuanced behavior within each area where variables can have different weightings and different factor scores based upon the dynamics of those units.

Jeff Terry
So just to make sure, so it's not a single model that's taking inputs for every unit and doing a forecast. It's actually a model for every unit.

Andy Day
Yeah. It's a system of models, one for every unit and then models at the roll-up levels as well. Yep.

Jeff Terry
Brilliant. And what are some of the inputs that play into a model like that?

Andy Day
Yeah, there's an array. We basically look at, as I mentioned earlier, every patient that's in the ED and where they are in the workflow and what we know right now, all patients that are in the OR schedule and where they are in their surgical workflow, all the patients that are already in bed and where they are in their care pathway in terms of what's the dynamics of that patient's individual situation is. We do all that with real time data and rerun the model. And we run those, for example, in that case, we rerun that model in learning mode every 48 hours for census, but we rerun the model dynamically on the hour, every hour. So as the day plays out and things move, things change, it's continuously re-forecasting. So it's not a static, one-time model.

Jeff Terry
So that's census forecast reruns every hour, trains every 48 hours. And I'd imagine that the training is particularly useful in a period like now with a pandemic, where things are changing in sort of unexpected ways, but the training algorithms will pick that up to some degree every 48 hours.

Andy Day
Yeah, absolutely. The models continuously relearn, right? And so, they will adapt over time. Now, if there's a big structural change that sort of causes a step function change, the learning mode will catch it. But what will also happen to ensure accuracy, we have something that monitors for drift. Basically it will send us an alert that says, "Hey, your accuracy has been going along in range where it should be, and then all of a sudden it falls off." That will alert us that maybe something big has changed, and we can go in and tune the model. Perhaps we need to make a structural change to the way the model's built.

Jeff Terry
Got it. And so, that census forecast and a great insight there. Another one that's very interesting, I find, is where we're predicting PACU and postop arrival patterns, and we're actually prescribing the placement of patients into postop bays based on the capabilities of the bays structure and the needs of the patient.

Andy Day
Yeah.

Jeff Terry
Can you comment on how that one works?

Andy Day
Yeah, sure. So we are forecasting the arrival times and the durations of each of the steps in the surgical process step. And then from that, we actually build a heuristic. That heuristic that takes inputs, which are those forecasts, so that we can basically model the dynamics of the surgical system in near real time. And so, that forecast basically rolls up and says, "Okay, here's what's most likely to happen." And then we use a mathematical model, a MIP, mixed integer programming model, to say which patient should actually get to the next bed, and which bed should it be to optimize for that patient's needs. So this is basically a system of models of which machine learning is an input.

Jeff Terry
That's good. I should have mentioned on that one, obviously the human makes the decision, but what I think this method does is it makes it much easier for them to make a decision. So they're making faster decision with less sort of cognitive load on them. And I believe we've found that the human decision math sort of takes that prescription more than 80% of the time.

Andy Day
That's right. More than 80% of the time the recommendation is taken verbatim. There are situations which are in the data. That's why we still want humans in the loop, situations that the data can't tell you what to do, the data that's available electronically in the system that we're forecasting for them. But yeah, it's worked quite well. And our accuracy, for example, of predicting wheels in time for a case late this afternoon is like plus or minus 15 minutes, which is much better than just what's in the schedule.

Jeff Terry
Which is remarkable really. Yeah. Well, Andy, that's great. And that notion of serving the human and keeping the human in the loop is probably a good place to close because that's what we're all about. This is serving caregivers never replacing caregivers. And so, thank you, Andy. Brilliant stuff as always. And with that, we'll close the podcast. Thanks to our audience.

Andy Day
Take care, folks.

12:28 min
#44 - Medication Management with Chief Medical Officer Dr. Colin Banas with DrFirst Cover Image
Making medication management frictionless helps health systems prevent negative outcomes, ...
Oct 11 · 12:28 min
Making medication management frictionless helps health systems prevent negative outcomes, adverse drug events and ...
Oct 11 · 12:28 min
12:50 min
#43 - Dosis Co-Founder and CEO Shiv Chhabra on Using AI to Transform Chronic Disease Management Cover Image
In this podcast Jeff Terry is joined by Shiv Chhabra, the CEO and founder of Dosis, which ...
Sep 22 · 12:50 min
In this podcast Jeff Terry is joined by Shiv Chhabra, the CEO and founder of Dosis, which is a company that helps get ...
Sep 22 · 12:50 min
14:48 min
#42 - User Interface with Ensues' Co-Founder and CEO Mitra Sangroula Cover Image
In this podcast, Jeff Terry, is joined by Mitra Sangroula, the founder and CEO of Ensue, ...
Sep 2 · 14:48 min
In this podcast, Jeff Terry, is joined by Mitra Sangroula, the founder and CEO of Ensue, which is a UI company. They’ll ...
Sep 2 · 14:48 min
20:09 min
#41 - Qventus CEO Mudit Garg on Real Time Healthcare Cover Image
In this podcast episode, Jeff Terry is joined by Mudit Garg, the CEO of Qventus. They’ll ...
Aug 9 · 20:09 min
In this podcast episode, Jeff Terry is joined by Mudit Garg, the CEO of Qventus. They’ll be discussing the vision for ...
Aug 9 · 20:09 min
15:23 min
#40 - Clinical Deterioration with Bradford Teaching Hospitals NHS Foundation Trust’s Deputy MD, LeeAnne Elliott Cover Image
In the next installment of the Real Time Healthcare podcast, Becky Knee is joined by ...
Aug 2 · 15:23 min
In the next installment of the Real Time Healthcare podcast, Becky Knee is joined by LeeAnne Elliot, Deputy Medical ...
Aug 2 · 15:23 min
19:22 min
#39 - Clinical Deterioration with Bradford Teaching Hospitals NHS Foundation Trust’s Sepsis Nurse Specialist, Clare Nandha Cover Image
In the next installment of the Real Time Healthcare podcast, Becky Knee is joined by ...
Jul 21 · 19:22 min
In the next installment of the Real Time Healthcare podcast, Becky Knee is joined by Clare Nandha, Lead Sepsis Nurse at ...
Jul 21 · 19:22 min
35:00 min
#38 - CEO Barb Collins on Clinical Apps for Safety & Quality at Humber River Hospital Cover Image
In this podcast, Jeff Terry talks to Barb Collins about Generation 2 of Humber River's ...
Jul 13 · 35:00 min
In this podcast, Jeff Terry talks to Barb Collins about Generation 2 of Humber River's command center. They’ll cover a ...
Jul 13 · 35:00 min
21:07 min
#37 - Perinatal Care with Humber River Hospital's Dr. Andre LaRoche and Jhanvi Solanki Cover Image
In this podcast episode, Jeff Terry is joined by Dr. Andre LaRoche and Jhanvi Solanki ...
Jun 15 · 21:07 min
In this podcast episode, Jeff Terry is joined by Dr. Andre LaRoche and Jhanvi Solanki from Humber River Hospital. ...
Jun 15 · 21:07 min

Subscribe to Podcast Updates: