THE REAL TIME HEALTHCARE PODCAST · Feb 4 · 10:20 min

#28 - Andy Day, Real Time Forecasting with Machine Learning

In this Episode:

Andy Day and Jeff Terry provide examples for when AI is used in Forecasting. They cover examples including forecast census / staffing differential at a unit level, bed placement decisions, PACU arrivals / post-op bay placement and disposition / day of discharge. Andy and Jeff talk about the system of models that are used and re-run with real-time data to enable easier decisions for caregivers.

Jeff Terry
Hello and welcome. I'm Jeff Terry. Very happy to be joined today by Andy Day who's the chief technology officer of the command center team. Good morning, Andy.

Andy Day
Morning.

Jeff Terry
We're going to talk today about forecasting in real time using machine learning. Of course, we do that in Tiles for quite a few things to forecast census, staffing differential, PACU arrivals. It's very difficult to do in real time. We'll try to help you see what that means and how that's useful. So, Andy, to start us off, could you share a bit more about the types of things that we're forecasting?

Andy Day
Sure. Picture's worth a thousand words, I'll share my screen here so you can follow along. So a few examples of where we use AI and machine learning. We've been doing it for the last 10 plus years in healthcare. That forecast is used in analytic Tiles to help our users actually consume it in a meaningful way. So some examples, top left corner here, you see the census and staffing forecast Tile, where we're forecasting census at a unit level, look every hour going out for the next 48, as well as looking seven to 14 days out, so you can better set your staffing and adjust for situations that a forecast where your unit is going to be running out of beds or in trouble before it happens.

Andy Day
Top right, based upon all the patients competing for capacity within the system and your historical behavior over the last couple of years, who's most likely to win each of those contests. And where were they most likely actually get placed so that you can better adapt for what's likely to happen and to make more consistent decisions in bed placement. Bottom right, this is for a surgical daycare unit, which does both prep and stage two recovery for a large set of ORs and procedural suites. And we're forecasting both the supply and demand of both the bays and staff across the top, as well as forecasting at a patient level when a case might be disrupted in the future based upon the dynamics of the system and how they're playing out, before it happens so we can act upon it.

Andy Day
And then in the bottom left, for every patient that is in the house, forecasting what their most likely day of discharge is going to be, and their post-discharge disposition, are they going to go to a SNIF or an LTAC or go home with home health, et cetera. And so, these are just some good examples that we've done many more. As you mentioned in your introduction, we forecast OR holds before they happen so you can try to mitigate those situations. We forecast the supply and demand and wait times for support services in a lot of different places in the hospital, and a number of others

Jeff Terry
That's good. So just recapping, so upper left, we have census, forecasts and staffing differential. Upper right, we're forecasting the bed the patient is likely to be assigned. Lower right, we're predicting PACU arrivals and recommending post-op bay placement. And then lower left, we're forecasting disposition and day of discharge. Those are great examples.

Jeff Terry
Andy, something you mentioned I want to draw out. So unlike retrospective use of machine learning, where all the data is available, diagnosis, the lab results are all back, everything's there, when we're forecasting in real time, we don't necessarily have all the data, so we have to build the models based on what information we're likely to have in the moment. Can you say more about that?

Andy Day
Sure. Yeah. When you're doing machine learning for research purposes, or just typical retrospective data type approach, there's a lot of discussion around these things they call data lakes, right? Which is where you try to get as many data sources as possible, as large those data sets that you can get, with as many variables as you can identify, and focus on basically establishing common ontologies, curating the data as it's arriving, and trying to create basically an environment in which your data science team, your machine learning specialists, can go fishing. And what I mean by that is for a topic of interest, can we find an algorithm which is helpful?

Andy Day
And so, this works well for when you're doing sort of, like you said, retrospective historical data mining and large data sets. But when you're trying to do real time forecasting, this methodology really doesn't work because you've got a patient in the ED, they're being worked up, they have some data in, they're missing a lot of data that hasn't come back yet, and there's a lot of other things you won't know that you will know post discharge when everything's settled. So this methodology really doesn't work for real time because it's all about data latency.

Jeff Terry
Got it. And so, we've figured out, you and I were joking earlier, it's almost like instead of a data lake, we figured out what tributaries feed the lake and where should we fish? What's the best place to fish on the tributary so we get the information that we need when we need it?

Andy Day
Yeah. I'd even say we designed the tributary and build it to just [inaudible 00:04:53] because any more water will slow things down.

Jeff Terry
That's great. So if you would, take one example, maybe census forecast, which I think is what we do more than, or have done the longest. How do we do it? How does it work?

Andy Day
Sure. So when census forecast, for example, we happen to use machine learning technology called Spark, Apache Spark. And we use random [inaudible 00:05:18] amplification because we found that method to be the most helpful, but we also use other methods embedded. And one thing we found to be really important, we do a lot of that, that's called ensemble modeling where you combine multiple models in one model, and we do a lot of that. We also have found that it's important to model every unit at the unit level, having one model that runs for everything, spreads the mayonnaise, doesn't reflect the nuanced behavior within each area where variables can have different weightings and different factor scores based upon the dynamics of those units.

Jeff Terry
So just to make sure, so it's not a single model that's taking inputs for every unit and doing a forecast. It's actually a model for every unit.

Andy Day
Yeah. It's a system of models, one for every unit and then models at the roll-up levels as well. Yep.

Jeff Terry
Brilliant. And what are some of the inputs that play into a model like that?

Andy Day
Yeah, there's an array. We basically look at, as I mentioned earlier, every patient that's in the ED and where they are in the workflow and what we know right now, all patients that are in the OR schedule and where they are in their surgical workflow, all the patients that are already in bed and where they are in their care pathway in terms of what's the dynamics of that patient's individual situation is. We do all that with real time data and rerun the model. And we run those, for example, in that case, we rerun that model in learning mode every 48 hours for census, but we rerun the model dynamically on the hour, every hour. So as the day plays out and things move, things change, it's continuously re-forecasting. So it's not a static, one-time model.

Jeff Terry
So that's census forecast reruns every hour, trains every 48 hours. And I'd imagine that the training is particularly useful in a period like now with a pandemic, where things are changing in sort of unexpected ways, but the training algorithms will pick that up to some degree every 48 hours.

Andy Day
Yeah, absolutely. The models continuously relearn, right? And so, they will adapt over time. Now, if there's a big structural change that sort of causes a step function change, the learning mode will catch it. But what will also happen to ensure accuracy, we have something that monitors for drift. Basically it will send us an alert that says, "Hey, your accuracy has been going along in range where it should be, and then all of a sudden it falls off." That will alert us that maybe something big has changed, and we can go in and tune the model. Perhaps we need to make a structural change to the way the model's built.

Jeff Terry
Got it. And so, that census forecast and a great insight there. Another one that's very interesting, I find, is where we're predicting PACU and postop arrival patterns, and we're actually prescribing the placement of patients into postop bays based on the capabilities of the bays structure and the needs of the patient.

Andy Day
Yeah.

Jeff Terry
Can you comment on how that one works?

Andy Day
Yeah, sure. So we are forecasting the arrival times and the durations of each of the steps in the surgical process step. And then from that, we actually build a heuristic. That heuristic that takes inputs, which are those forecasts, so that we can basically model the dynamics of the surgical system in near real time. And so, that forecast basically rolls up and says, "Okay, here's what's most likely to happen." And then we use a mathematical model, a MIP, mixed integer programming model, to say which patient should actually get to the next bed, and which bed should it be to optimize for that patient's needs. So this is basically a system of models of which machine learning is an input.

Jeff Terry
That's good. I should have mentioned on that one, obviously the human makes the decision, but what I think this method does is it makes it much easier for them to make a decision. So they're making faster decision with less sort of cognitive load on them. And I believe we've found that the human decision math sort of takes that prescription more than 80% of the time.

Andy Day
That's right. More than 80% of the time the recommendation is taken verbatim. There are situations which are in the data. That's why we still want humans in the loop, situations that the data can't tell you what to do, the data that's available electronically in the system that we're forecasting for them. But yeah, it's worked quite well. And our accuracy, for example, of predicting wheels in time for a case late this afternoon is like plus or minus 15 minutes, which is much better than just what's in the schedule.

Jeff Terry
Which is remarkable really. Yeah. Well, Andy, that's great. And that notion of serving the human and keeping the human in the loop is probably a good place to close because that's what we're all about. This is serving caregivers never replacing caregivers. And so, thank you, Andy. Brilliant stuff as always. And with that, we'll close the podcast. Thanks to our audience.

Andy Day
Take care, folks.

12:20 min
#29 - Jen Naylor, Observation Pt Expediter at Tampa General Cover Image
In this podcast episode Jeff Terry, CEO Command Centers, GE Healthcare interviews Jen ...
Feb 8 · 12:20 min
In this podcast episode Jeff Terry, CEO Command Centers, GE Healthcare interviews Jen Naylor, Managing Director, GE ...
Feb 8 · 12:20 min
10:20 min
#28 - Andy Day, Real Time Forecasting with Machine Learning Cover Image
Andy Day and Jeff Terry provide examples for when AI is used in Forecasting. They cover ...
Feb 4 · 10:20 min
Andy Day and Jeff Terry provide examples for when AI is used in Forecasting. They cover examples including forecast ...
Feb 4 · 10:20 min
18:59 min
#27 - Ian Worden, MBA, MS, MA, HFMA, CPA, System Vice President and Chief Operating Officer, Virginia Mason Franciscan Health. Cover Image
Join us for a podcast conversation between Jeff Terry, GE Healthcare and Ian Worden, ...
Feb 1 · 18:59 min
Join us for a podcast conversation between Jeff Terry, GE Healthcare and Ian Worden, System Vice President and Chief ...
Feb 1 · 18:59 min
17:34 min
#26 - Mary Alice Ragsdale, MSN, MBA, RN VP Operations COO, St. Anthony Hospital COO, Mission Control & Virtual Health Services​ Cover Image
Interview with Mary Ragsdale, COO of Virginia Mason Franciscan Health, St. Anthony’s ...
Jan 27 · 17:34 min
Interview with Mary Ragsdale, COO of Virginia Mason Franciscan Health, St. Anthony’s Hospital and Mission Control. Mary ...
Jan 27 · 17:34 min
20:49 min
#25 - Jim Scheulen, CAO, Johns Hopkins Capacity Command Center Cover Image
On this podcast, Jeff is joined by Jim Scheulen, Chief Administrative Officer of Capacity ...
Jan 21 · 20:49 min
On this podcast, Jeff is joined by Jim Scheulen, Chief Administrative Officer of Capacity Management and Emergency ...
Jan 21 · 20:49 min
11:37 min
#24 - Gabriella Devine, Treatment Team Workload at OHSU Cover Image
Jeff Terry and Gabriella Devine discuss how OHSU Mission Control implemented Treatment ...
Jan 13 · 11:37 min
Jeff Terry and Gabriella Devine discuss how OHSU Mission Control implemented Treatment Team Workload Tile, a new app ...
Jan 13 · 11:37 min
21:16 min
#23 - Meg Rivera, Tamas Fixler, Patient Manager at AdventHealth Cover Image
This episode describes the use, configuration and deployment of Patient Manager Tile at ...
Dec 22 · 21:16 min
This episode describes the use, configuration and deployment of Patient Manager Tile at eight AdventHealth hospitals in ...
Dec 22 · 21:16 min
11:01 min
#22 - Patient Manager Tile - A Personal Command Center for Caregivers Cover Image
This is Patient Manager, the Tile that puts a command center at every caregiver’s ...
Oct 29 · 11:01 min
This is Patient Manager, the Tile that puts a command center at every caregiver’s fingertips. Patient Manager helps ...
Oct 29 · 11:01 min
04:39 min
#21 - Patient Manager Tile - Profiles and Search Cover Image
It's so important that users can work at speed in the Patient Manager Tile. This video ...
Oct 15 · 04:39 min
It's so important that users can work at speed in the Patient Manager Tile. This video shows a bit about profiles and ...
Oct 15 · 04:39 min
04:19 min
#20 - Patient Manager Tile with Downgrade Module Cover Image
Who is ready and eligible for downgrade to another level of care? All information at once.
Oct 5 · 04:19 min
Who is ready and eligible for downgrade to another level of care? All information at once.
Oct 5 · 04:19 min
05:52 min
#19 - Advanced Illness Tile Cover Image
The Real Time all-the-time Tile helps to manage the cohort of patients who have advanced ...
Sep 28 · 05:52 min
The Real Time all-the-time Tile helps to manage the cohort of patients who have advanced illness and may be planning ...
Sep 28 · 05:52 min
07:02 min
#18 - Patient Manager Tile Introduction Cover Image
The Patient Manager Tile a game changer for two reasons: it enables multidisciplinary ...
Jul 14 · 07:02 min
The Patient Manager Tile a game changer for two reasons: it enables multidisciplinary teams to problem solve in Real ...
Jul 14 · 07:02 min
06:25 min
#17 - Procedural Snapshot Tile Cover Image
The Procedural Snapshot Tile provides situational awareness as well as patient-level ...
Jul 9 · 06:25 min
The Procedural Snapshot Tile provides situational awareness as well as patient-level actionable alerts to help keep ...
Jul 9 · 06:25 min
06:34 min
#16 - Obstetrics and Neonatal Tile Cover Image
Following an evidence-based standard of care, the Obstetrics and Neonatal Tiles improve ...
Jul 7 · 06:34 min
Following an evidence-based standard of care, the Obstetrics and Neonatal Tiles improve the situational awareness for ...
Jul 7 · 06:34 min
06:14 min
#15 - Anesthesia Exceptions Tile Cover Image
The Anesthesia Exceptions Tile is used to coordinate and assist the flow of complex ...
Jun 30 · 06:14 min
The Anesthesia Exceptions Tile is used to coordinate and assist the flow of complex procedural cases across an array of ...
Jun 30 · 06:14 min
07:37 min
#14 - ED Expediter Tile Cover Image
The ED Expediter Tile is useful in advancing patients through emergency departments and ...
Jun 25 · 07:37 min
The ED Expediter Tile is useful in advancing patients through emergency departments and providing patient level alerts ...
Jun 25 · 07:37 min
06:02 min
#13 - Transfers Tile Cover Image
The Transfers Tile aggregates information on the individual patients queued to transfer ...
Jun 23 · 06:02 min
The Transfers Tile aggregates information on the individual patients queued to transfer into the system.
Jun 23 · 06:02 min
14:18 min
#12 - Discharge Barriers Tile Cover Image
The Discharge Barriers Tile outlines the key actions the care team can take to facilitate ...
Jun 18 · 14:18 min
The Discharge Barriers Tile outlines the key actions the care team can take to facilitate patients who are identified ...
Jun 18 · 14:18 min
10:31 min
#11 - Boarders Tile Cover Image
The Boarders Tile provides facility and system level visibility to all inbound pressure ...
Jun 16 · 10:31 min
The Boarders Tile provides facility and system level visibility to all inbound pressure on the system to take patients ...
Jun 16 · 10:31 min
14:43 min
#10 - Capacity Snapshot Tile Cover Image
The Capacity Snapshot Tile provides insight on bed capacity both in the moment and after ...
Jun 11 · 14:43 min
The Capacity Snapshot Tile provides insight on bed capacity both in the moment and after we process decisions.
Jun 11 · 14:43 min
07:36 min
#09 - Critical Resources Tile Cover Image
From State-level to Unit-level, Real Time awareness of available beds.
Jun 9 · 07:36 min
From State-level to Unit-level, Real Time awareness of available beds.
Jun 9 · 07:36 min
06:46 min
#08 - Infectious Disease Tile Cover Image
The Infectious Disease Tile was developed in response the COVID-19 pandemic. It provides ...
Jun 4 · 06:46 min
The Infectious Disease Tile was developed in response the COVID-19 pandemic. It provides visualization of patient load ...
Jun 4 · 06:46 min
05:08 min
#07 - Post Acute Placement Tile Cover Image
The Post Acute Placement Tile is used to reduce excess days and facilitate outplacement ...
Jun 2 · 05:08 min
The Post Acute Placement Tile is used to reduce excess days and facilitate outplacement of patients from an inpatient ...
Jun 2 · 05:08 min
06:49 min
#06 - Imaging Expediter Tile Cover Image
The Imaging Expediter Tile provides Real Time visibility to available capacity by ...
May 28 · 06:49 min
The Imaging Expediter Tile provides Real Time visibility to available capacity by modality and supports operational ...
May 28 · 06:49 min
04:20 min
#05 - Care Progression Tile Cover Image
The Care Progression Tile provides hospitals the ability to track patients throughout ...
May 26 · 04:20 min
The Care Progression Tile provides hospitals the ability to track patients throughout their inpatient stay.
May 26 · 04:20 min
05:15 min
#04 - Risk of Harm Tile Cover Image
The Risk of Harm Tile provides visibility to track variations in practice versus hospital ...
May 21 · 05:15 min
The Risk of Harm Tile provides visibility to track variations in practice versus hospital defined care plans.
May 21 · 05:15 min
05:51 min
#03 - Seniors Care Tile Cover Image
The Seniors Care Tile helps ensure geriatric inpatients outside the geriatric unit ...
May 19 · 05:51 min
The Seniors Care Tile helps ensure geriatric inpatients outside the geriatric unit receive best practice care in ...
May 19 · 05:51 min
08:46 min
#02 - Discharge Planning and Escalation Tile Cover Image
The Discharge Planning Tile empowers rounding across the system by providing a fast way ...
May 14 · 08:46 min
The Discharge Planning Tile empowers rounding across the system by providing a fast way to review patients, log ...
May 14 · 08:46 min
06:34 min
#01 - Clinical Deterioration Tile Cover Image
The Clinical Deterioration Tile brings real-time, all-the-time awareness of the status of ...
Aug 1 · 06:34 min
The Clinical Deterioration Tile brings real-time, all-the-time awareness of the status of the most critically ill ...
Aug 1 · 06:34 min

Subscribe to Podcast Updates: