#28 - Andy Day, Real Time Forecasting with Machine Learning

Feb 4 · 10:20 min

In this Episode:

Andy Day and Jeff Terry provide examples for when AI is used in Forecasting. They cover examples including forecast census / staffing differential at a unit level, bed placement decisions, PACU arrivals / post-op bay placement and disposition / day of discharge. Andy and Jeff talk about the system of models that are used and re-run with real-time data to enable easier decisions for caregivers.

Jeff Terry
Hello and welcome. I'm Jeff Terry. Very happy to be joined today by Andy Day who's the chief technology officer of the command center team. Good morning, Andy.

Andy Day
Morning.

Jeff Terry
We're going to talk today about forecasting in real time using machine learning. Of course, we do that in Tiles for quite a few things to forecast census, staffing differential, PACU arrivals. It's very difficult to do in real time. We'll try to help you see what that means and how that's useful. So, Andy, to start us off, could you share a bit more about the types of things that we're forecasting?

Andy Day
Sure. Picture's worth a thousand words, I'll share my screen here so you can follow along. So a few examples of where we use AI and machine learning. We've been doing it for the last 10 plus years in healthcare. That forecast is used in analytic Tiles to help our users actually consume it in a meaningful way. So some examples, top left corner here, you see the census and staffing forecast Tile, where we're forecasting census at a unit level, look every hour going out for the next 48, as well as looking seven to 14 days out, so you can better set your staffing and adjust for situations that a forecast where your unit is going to be running out of beds or in trouble before it happens.

Andy Day
Top right, based upon all the patients competing for capacity within the system and your historical behavior over the last couple of years, who's most likely to win each of those contests. And where were they most likely actually get placed so that you can better adapt for what's likely to happen and to make more consistent decisions in bed placement. Bottom right, this is for a surgical daycare unit, which does both prep and stage two recovery for a large set of ORs and procedural suites. And we're forecasting both the supply and demand of both the bays and staff across the top, as well as forecasting at a patient level when a case might be disrupted in the future based upon the dynamics of the system and how they're playing out, before it happens so we can act upon it.

Andy Day
And then in the bottom left, for every patient that is in the house, forecasting what their most likely day of discharge is going to be, and their post-discharge disposition, are they going to go to a SNIF or an LTAC or go home with home health, et cetera. And so, these are just some good examples that we've done many more. As you mentioned in your introduction, we forecast OR holds before they happen so you can try to mitigate those situations. We forecast the supply and demand and wait times for support services in a lot of different places in the hospital, and a number of others

Jeff Terry
That's good. So just recapping, so upper left, we have census, forecasts and staffing differential. Upper right, we're forecasting the bed the patient is likely to be assigned. Lower right, we're predicting PACU arrivals and recommending post-op bay placement. And then lower left, we're forecasting disposition and day of discharge. Those are great examples.

Jeff Terry
Andy, something you mentioned I want to draw out. So unlike retrospective use of machine learning, where all the data is available, diagnosis, the lab results are all back, everything's there, when we're forecasting in real time, we don't necessarily have all the data, so we have to build the models based on what information we're likely to have in the moment. Can you say more about that?

Andy Day
Sure. Yeah. When you're doing machine learning for research purposes, or just typical retrospective data type approach, there's a lot of discussion around these things they call data lakes, right? Which is where you try to get as many data sources as possible, as large those data sets that you can get, with as many variables as you can identify, and focus on basically establishing common ontologies, curating the data as it's arriving, and trying to create basically an environment in which your data science team, your machine learning specialists, can go fishing. And what I mean by that is for a topic of interest, can we find an algorithm which is helpful?

Andy Day
And so, this works well for when you're doing sort of, like you said, retrospective historical data mining and large data sets. But when you're trying to do real time forecasting, this methodology really doesn't work because you've got a patient in the ED, they're being worked up, they have some data in, they're missing a lot of data that hasn't come back yet, and there's a lot of other things you won't know that you will know post discharge when everything's settled. So this methodology really doesn't work for real time because it's all about data latency.

Jeff Terry
Got it. And so, we've figured out, you and I were joking earlier, it's almost like instead of a data lake, we figured out what tributaries feed the lake and where should we fish? What's the best place to fish on the tributary so we get the information that we need when we need it?

Andy Day
Yeah. I'd even say we designed the tributary and build it to just [inaudible 00:04:53] because any more water will slow things down.

Jeff Terry
That's great. So if you would, take one example, maybe census forecast, which I think is what we do more than, or have done the longest. How do we do it? How does it work?

Andy Day
Sure. So when census forecast, for example, we happen to use machine learning technology called Spark, Apache Spark. And we use random [inaudible 00:05:18] amplification because we found that method to be the most helpful, but we also use other methods embedded. And one thing we found to be really important, we do a lot of that, that's called ensemble modeling where you combine multiple models in one model, and we do a lot of that. We also have found that it's important to model every unit at the unit level, having one model that runs for everything, spreads the mayonnaise, doesn't reflect the nuanced behavior within each area where variables can have different weightings and different factor scores based upon the dynamics of those units.

Jeff Terry
So just to make sure, so it's not a single model that's taking inputs for every unit and doing a forecast. It's actually a model for every unit.

Andy Day
Yeah. It's a system of models, one for every unit and then models at the roll-up levels as well. Yep.

Jeff Terry
Brilliant. And what are some of the inputs that play into a model like that?

Andy Day
Yeah, there's an array. We basically look at, as I mentioned earlier, every patient that's in the ED and where they are in the workflow and what we know right now, all patients that are in the OR schedule and where they are in their surgical workflow, all the patients that are already in bed and where they are in their care pathway in terms of what's the dynamics of that patient's individual situation is. We do all that with real time data and rerun the model. And we run those, for example, in that case, we rerun that model in learning mode every 48 hours for census, but we rerun the model dynamically on the hour, every hour. So as the day plays out and things move, things change, it's continuously re-forecasting. So it's not a static, one-time model.

Jeff Terry
So that's census forecast reruns every hour, trains every 48 hours. And I'd imagine that the training is particularly useful in a period like now with a pandemic, where things are changing in sort of unexpected ways, but the training algorithms will pick that up to some degree every 48 hours.

Andy Day
Yeah, absolutely. The models continuously relearn, right? And so, they will adapt over time. Now, if there's a big structural change that sort of causes a step function change, the learning mode will catch it. But what will also happen to ensure accuracy, we have something that monitors for drift. Basically it will send us an alert that says, "Hey, your accuracy has been going along in range where it should be, and then all of a sudden it falls off." That will alert us that maybe something big has changed, and we can go in and tune the model. Perhaps we need to make a structural change to the way the model's built.

Jeff Terry
Got it. And so, that census forecast and a great insight there. Another one that's very interesting, I find, is where we're predicting PACU and postop arrival patterns, and we're actually prescribing the placement of patients into postop bays based on the capabilities of the bays structure and the needs of the patient.

Andy Day
Yeah.

Jeff Terry
Can you comment on how that one works?

Andy Day
Yeah, sure. So we are forecasting the arrival times and the durations of each of the steps in the surgical process step. And then from that, we actually build a heuristic. That heuristic that takes inputs, which are those forecasts, so that we can basically model the dynamics of the surgical system in near real time. And so, that forecast basically rolls up and says, "Okay, here's what's most likely to happen." And then we use a mathematical model, a MIP, mixed integer programming model, to say which patient should actually get to the next bed, and which bed should it be to optimize for that patient's needs. So this is basically a system of models of which machine learning is an input.

Jeff Terry
That's good. I should have mentioned on that one, obviously the human makes the decision, but what I think this method does is it makes it much easier for them to make a decision. So they're making faster decision with less sort of cognitive load on them. And I believe we've found that the human decision math sort of takes that prescription more than 80% of the time.

Andy Day
That's right. More than 80% of the time the recommendation is taken verbatim. There are situations which are in the data. That's why we still want humans in the loop, situations that the data can't tell you what to do, the data that's available electronically in the system that we're forecasting for them. But yeah, it's worked quite well. And our accuracy, for example, of predicting wheels in time for a case late this afternoon is like plus or minus 15 minutes, which is much better than just what's in the schedule.

Jeff Terry
Which is remarkable really. Yeah. Well, Andy, that's great. And that notion of serving the human and keeping the human in the loop is probably a good place to close because that's what we're all about. This is serving caregivers never replacing caregivers. And so, thank you, Andy. Brilliant stuff as always. And with that, we'll close the podcast. Thanks to our audience.

Andy Day
Take care, folks.

As a rapidly growing health system serving a wide geographical and largely rural area, Deaconess needed a way to ...
Serving a large geographical area across four states, Deaconess Health System is evolving their telehealth programs to ...
As an experienced charge nurse for a fast-paced Medicine unit with complex, high-acuity patients, Vicky Martin and her ...
We must sustain our caregivers if we’re going to achieve our other sustainability goals. See how leading health systems ...
Seth Sternberg and his team at Honor are committed to changing the aging experience. For them, that starts with caring ...
Matthew Mawby is a nurse and co-founder of StaffHealth, and he's working to take the hassle out of nurse staffing. Jeff ...
As the Regional Director of Capacity and Throughput for the Providence Health System in Oregon, Helene Anderson has ...
Leveraging technology already in place at Oregon Science & Health University, Apprise Health Insights partnered ...

Subscribe to Podcast Updates: