Artificial intelligence (AI) has been an emerging hot topic over the last several months with the rise of Open AI’s ChatGPT, Microsoft’s integration of ChatGPT technology into its Bing search engine, and Google’s announcement of its own chatbot, known as Bard. And while there are concerns about the more “general AI” technologies built to improve neural network capabilities so they are comparable to those of humans, health care systems are able to expand their services by leveraging the more familiar “narrow” or single-task AI tools, such as virtual chat assistance. Deploying this kind of AI technology can lead to an enhanced self-service experience for patients.
We welcome Matt White, Director of Innovation at Contra Costa Health Services (CCHS), who shares how they’ve begun to thoughtfully integrate AI technology in order to better understand their patient engagement, with the ultimate aim to provide a consistent experience across all digital channels.
Listen and subscribe to CCI’s Health Pilots podcast on Apple Podcasts, Spotify, and elsewhere. Below is a transcript of the episode, edited for readability.
Episode Teaser // Matt White, Contra Costa Health Services (CCHS):
…We’re really, I think, aware of the negative interactions people have already had with bots. And so we were just really thoughtful about how we deployed it. And so just making sure that we were offering this as an option for people to use, not something they had to do. And then we were very thoughtful at the end of the interaction to say, “If we didn’t answer your questions, we were still happy to pass you through to an agent.” And we think that some of that really human-centered design experience really kind of played into the deploying the technology in a responsible way that was received well by our community.
Health Pilots podcast intro:
Welcome to the Health Pilots podcast presented by the Center for Care Innovations. This podcast is about strengthening the health and wellbeing of historically underinvested communities. Every episode offers new ideas and practical advice that you can apply today!
Jessica Ortiz, Center for Care Innovations (CCI):
Hi everyone. I’m Jessica Ortiz with the Center for Care Innovations. Today we’re talking about one of the exciting projects that Contra Costa Health Services has been working on for the past two and a half years. Our hope is that by sharing the highs and lows of this digital health solution, safety net organizations can apply the lessons learned to their own challenges. I’m here with Matt White, member of our Tech Hub Learning Network, which is 14 tech-forward California based community health centers, clinic coalitions, and primary care departments, and county health systems that are working to accelerate the adoption of innovative technologies. We partner with our Tech Hub members to vet, pilot, evaluate, and spread innovative digital health solutions, serving Medicaid markets, and historically under invested communities. We are very excited to bring you the story today. Matt, can you introduce yourself to the listeners, share a little bit about you and your organization?
Matt White, CCHS:
Sure. My name is Matt White, and Contra Costa Health Services is a government health county-based system, primarily serving the Medicaid population in the eastern San Francisco Bay Area with approximately 1.2(ish) million residents. We also have a provider delivery network health plan as well as EMS, hazmat, environmental health, public health – so kind of an integrated health safety net system. And I am part of our Office and Informatics group, so I focus a lot on data and innovation, and technology as well as human-centered design and use those skills to try and bring some new services and solutions into our system.
Jessica Ortiz, CCI:
Great. Thanks, Matt. And we’re really happy to have you. So excited to hear about the project you’ve been working on. And just going right into that, can you tell us more about the project? How did you start?
Matt White, CCHS:
Well, I think COVID happened, I think is the truth of it. This vendor we actually had met with prior to COVID, and I thought they had a really interesting solution, and I’d looked at a number of different kind of chat digital assistants over a year or so prior to COVID, and I just saw the amount of work that was really required to do much with them was really more than we could really take on and support. So, this vendor, Hyro, who I had looked at previously when COVID happened, and I was actually reconnected to them as part of the CCI community, so I kind of reengaged. And obviously things had changed with COVID, we needed solutions, we needed them fast, and they had a free COVID product that they were already using at a number of other health systems. And all of a sudden, we went from having hundreds of web visitors a day to having tens of thousands of visitors a day, and really needed to think about solutions that could scale and interact with a community that we hadn’t really engaged with in that way before.
So, it was good timing, good product, and kind of a new need for us that all kind of came together at the start of COVID, and became one of the kind of, I think success stories we had with taking advantage of an opportunity to introduce some new technologies to the health system. And it didn’t come without its challenges or doubters. And so, we kind of had to grow into this as an organization, and we’re still learning. I think some of the best ways to deploy this. We try and phrase things as far as an opportunity or challenge statement – this thing just also happened I think at the right time. There’s definitely been a little bit more focus on defining the problem as we’ve moved along with this partner. But to start with, it was more of a “How do we deflect these tens of thousands of calls a day to help people find the information and answers that they need from a trusted source, and not have to call a kind of hastily put together call center around the COVID response? So, “how can we introduce some of these ideas around technology, self-service, kind of digital assistance, in a way that didn’t offend people or create frustration?” Which, I think a lot of us have experienced with some of these types of products previously.
Matt White, CCHS:
So, as we started to test these assumptions around the usage of a tool like this, we saw a lot of really positive feedback, a lot of really successful interactions through our website, which was a relatively new channel for us to do this kind of interactive digital space for people. So essentially people would come to the website, there’d be a little popup that people would see that said, “Hi. I’m a digital assistant. How could I help you?” And we tried to be really clear with people what we could help with. We didn’t want people to get frustrated thinking it could do a bunch of things, and it wasn’t really kind of trained to do so. Some of the nice things about this product was we could point it towards different types of content that it would scrape daily.
So, as we would update information around COVID, we didn’t have to go do something else to update this digital assistant. It would just kind of happen automatically, and those topics would be available for people to search. And so, it also allowed us to keep a pulse on COVID as things would continue to change. We were able to use data and analytics to help us add additional features to the product, for example, where to find testing when testing was really challenging. Where to find vaccines when vaccines were really challenging, and how to continue to help as the information changes. From this experience we saw with COVID, and I think it was kind of an early indicator of some of the staffing challenges that were to come with people getting moved around, and reallocated, and burnt out, and moved into different roles. And so, we also really saw this as an opportunity to help with automation in using this product to help us replace staff that were potentially allocated towards the public health emergency.
So, an example of this is, as part of our testing and vaccine strategy, we had a large influx of people that were getting access to MyChart, which is our patient portal. And people forget passwords, right? We’re human. And people would get locked out, and they would need help. And so, our small staff that previously supported those kinds of actions around usage and password resets, we were outsourcing it now to a third party, and every one of those phone calls was $20. And so, looking at the volume of password resets that we were now experiencing, there was a lot of cost that was going to it, and those agents weren’t really equipped to do anything more than direct people to websites to then reset the product, the password themselves. So, we thought that that would be an easy thing to automate. And so, this kind of became our first attempt using this digital assistant through the call center.
So, we’re really, I think, aware of the negative interactions people have already had with bots. And so, we were just really thoughtful about how we deployed it. And so just making sure that we were offering this as an option for people to use, not something they had to do. And then we were very thoughtful at the end of the interaction to say, “If we didn’t answer your questions, we were still happy to pass you through to an agent.” And we think that some of that really human-centered design experience really kind of played into the deploying the technology in a responsible way that was received well by our community. And what we saw pretty quickly was we were able to deflect 50%, 60% all the way up to 80% of the password reset calls by just basically offering to send a text message to somebody so that they could go to the website directly, and then reset the password themselves, rather than waiting on hold to talk to somebody that would essentially do the same thing.
And this came with a number of benefits. We could now offer self-service, offer service outside of hours of that call center. We allowed people to be empowered and to help to participate in their health, to learn about tools that maybe they hadn’t known were available before. So, I think there were a lot of really positive benefits that came from doing something like that.
And we even went and looked at the data and made sure that for the people that were calling, that they actually ended up getting their password reset, just to make sure we weren’t creating this extra frustration or friction where they would then have to call back and talk to somebody. So that was our early, I think “big win,” that kind of justified us looking at alternate use-cases for this company and product outside of COVID, and outside of some of that earlier solution that we had deployed. I just want to call out too, that the solution hasn’t been able to support languages to the extent that we would like, but we have done some interesting things that have allowed us to learn about people that speak Spanish, or Chinese, or other languages.
And so, what we’ve done is we’ve kind of deployed little suggestion buttons or options where it says, “Hey, if you speak Spanish, click here.” Or it can actually recognize that somebody’s typing Spanish, and we can at least present them with, “We can’t currently support this in this language, but call here.” And it’s allowed us to collect some really interesting data around who’s using our websites and digital tools, who would prefer to communicate with us in another language, which has allowed us to focus the resources and start to understand some of those things a little better, which we didn’t have before. And then hopefully with time we can start to develop a more complete solution that can support other languages. But this vendor has really allowed us to do some things that we wouldn’t have been able to support at all with other products in the way that they’re configured. So, it’s been a learning exercise, and I think we wanted to go with what we thought was the best product, even though it’s continuing to develop.
Jessica Ortiz, CCI:
That’s great, Matt. It’s nice to hear that you’re seeing some good outcomes. It’s interesting to see how you can collect some data to maybe make a case for expanding this product maybe to certain populations that speak different languages, that might be ready to access tech technology, but the support for translation is just not there yet, but hopefully will be in the future. You did earlier talk a little bit about how the platform scrubs information for you to keep that updated. What are some examples of content that’s being scrubbed and put into this platform?
Matt White, CCHS:
So, it was primarily around our website content. So, we tried to reuse things that we’d already kind of created. Primarily frequently asked questions around COVID became the basis for a lot of that knowledge. But we also are looking at expanding it to other types of groups within the health department. So, for example, if I’m a health services member and I want to look up information about my member benefits, currently that involves reading through a bunch of documents to see if things are covered or not. And we’re in the process of trying to convert that into information that could be accessed via the bot, or we’re calling it a “digital helper” / “digital assistant,” that people could also benefit from. And again, we came up with that by looking at what types of questions are people asking our “health plan”-bot and then letting that help guide us to the areas where we’re spending time.
So, another area has been around provider searches, so, “How can we help people find appropriate providers, and really improve upon that provider search ability?” So again, it’s this whole idea of using data to help drive what’s important for people that are visiting our websites, and then working with them to kind of think through what a solution could be. And we meet with them weekly. They work in two-week sprints. And so, we’ve found that we’ve been able to turn a lot of these things from ideas into reality within a couple of months, which has been pretty impressive.
Jessica Ortiz, CCI:
And how many different bots do you have that you’re running? Because you said you were talking about the “health plan”-bot. Are there different types of bots that you’re running on different topic areas?
Matt White, CCHS:
Yeah. So, we have the COVID-bot, we have the “health plan”-bot on the website. But one of the things we learned was very few people that use our health plan go to the website. So that was a really powerful learning for us to think about, “Where are people going? How are they interacting with us? And if we want to enhance our digital presence, what should that look like?” So we’ve actually reinvested in some of our website products, our digital tools, our outreach and communication – and we’re starting to use opportunities where people may call us to offer them services that they can access themselves now via the portal, via website, and help in that kind of digital transformation that people talk about, and what those digital front doors, and things like that really should look like.
But also needing to understand who your population is, what they’re using now, and then, “How do you help them access these newer solutions, and what does that support look like?” And I think those are the things that we’re kind of trying to figure out right now, but it’s been part of the kind of journey that we’ve gone on.
CCI Commercial:
The Telehealth Improvement Community Fund is working to expand access to video visits, namely for people who have experienced economic disparities, housing and food insecurity, institutional discrimination, or violence.
Join us as we continue the Accessible Video Visits series, powered by the Telehealth Improvement Community Fund. Our next webinar in the series, entitled “Set up for Success: Supporting Patients and Providers Through Video Visits” is on April 5th at 12pm Pacific. Learn promising practices from leading community health innovators in the field to enhance video visit delivery and how you can equip your clinic for a new era of virtual care. Register for the free webinar today by visiting careinnovations.org/events or visit the link in our show notes.
Jessica Ortiz, CCI:
I’m also wondering, earlier you were talking about using the bot to help folks with their password management. Folks that are forgetting their passwords, and it takes a lot of support to do that. How long did it take to go from that idea of, “Oh, we can use this for password reset”, to actually implementing that in practice at Contra Costa?
Matt White, CCHS:
I think it was, again, kind of done within a couple months. I really appreciate a company that can move quickly, and they also challenge us sometimes to think about what is the minimum viable product that we need to do something better than what we have, and to learn more as we go versus this tendency of spending six months or a year on something to try and get it, “right”, and then only to realize that half of what you did needs to be undone. So, it’s a different approach to working together, but it’s been enjoyable. And part of this work is, “How do you create the kind of culture that feels comfortable with doing that,” so that you can work well together.
Jessica Ortiz, CCI:
So Matt, could you share with the listeners any key results of your project so far? I know you mentioned some numbers earlier – that were a number of folks that were accessing this bot that you’re using. Is there anything else that you can share maybe about how this was received by staff at Contra Costa.
Matt White, CCHS:
I can share a few things. I mean, I think from an informational perspective, we always were sampling the conversations so we could actually see the specific conversations that happened with any kind of personal identifying information automatically removed. But we were able to at least see that often 90% or 100% of the sample basically received the best response that we would’ve wanted to provide, which was always kind of gave us a good level of confidence that the solution was working. And it also gave us opportunities to look for what we call, “missing knowledge”, or things that we could make better, which again is a really nice feedback loop to have. So, that was, early on, one of the things we looked for, also, our vendor was able to provide us with some benchmarks around what a good engagement rate looked like. We also kind of changed the intro or the topics to try and appeal to what we thought we could be most helpful with, which I think was useful.
And then we also would track conversion rates on various activities. So, if we knew that a certain path should result in some sort of expected action – “were we seeing that?” And so those were some things that we used to give us some confidence. I did mention earlier some of the work around the password reset and looking at the percentage of password resets that we were getting through Hyro versus our outsourced partner, and sampling to see that people were truly getting their password resets – was another area that we kind of collected some data and did some work. And then most recently, something that I’m pretty excited about is that our Member Services call center – we’ve done some analysis and looked at about a third of the total calls are around changing their provider. So, it’s been an area that we’re digging into, and we think we can make a really meaningful impact.
And part of that challenge is information about the providers live in different systems. So really thinking about how you can bring that together. And not to say that we didn’t have attempts to do that before, we had a web form, right? But like I said, people weren’t really used to our website, so people weren’t really going and looking for it. Then we added a message to the phone system saying, “While people are on hold, if you want, you can go to our website, “cchealth.org/healthplan,” and fill out the form.” There was a very minor increase in the number of people that dropped off the phone, went to the website, filled out the form. So, working with Hyro, we just said, “Hey, what’s the minimum change that we need to make to feel like we could really make an impact on this?” And so, we decided to say, “Instead of just telling people, drop off the phone and go to the website, we’re going to send you the link.
So, if you’re interested in having us send you the link to go to this website, we’ll do that now.” And, just by doing that small change, we were able to deflect 20-30% of people wanting to make a PCP (primary care physician) change. And while there’s still a lot of things that we’re investigating to improve those numbers, I think it’s just important to put yourself in that member’s or person’s journey, and think about the experience and go, “What could we do here to make this easier for somebody?” And then, do you get the expected result? And that’s allowed us to then say, “Okay. You know what? People want to do this. People don’t want to wait on hold, they want another option.” And then you can use those data points to really drive what that next action is. And we’ve got a number of things that we’re playing around with now to really try and make a difference.
Jessica Ortiz, CCI:
That leads me to think of, how is that working? So you have folks call into the call center and is there some sort of connection or integration with Hyro, where they press a button and then it triggers Hyro to send a message? Could you just touch on how that works a little bit more?
Matt White, CCHS:
Sure. Unfortunately, you still call in, and you get the IVR (interactive voice response) phone tree. We have dreams of replacing parts of that with some more kind of natural language. But for the moment, again, it’s starting small and looking for where you think you can make the biggest impact. There are ways to do more advanced levels of integration. So, we could say, for example, “If the hold time is less than two minutes, maybe we won’t even try and transfer the person, but if the hold time is 10 plus minutes, then we want them to get this.” So, there are ways of making it even more intelligent. But for the moment, we’ve started with just a basic call transfer. So, we ask somebody, “Is this the reason why you’re calling, and are you interested in having a digital tool help you with this?” And if they say, “Yes”, then we say, “Is this the phone number that you’re calling from that we can send a text message to you to help you complete this change?”
And if they say, “Yes”, then we send it, they say, “No”, then we get the appropriate information, and then we send it. And then we still ask at the end, “Is there anything else that you would like us to help you with so that we can get you back to an agent?” So, we don’t just assume that that’s the end of the interaction. So, we’ve tried to deploy in a way where we tell them what to expect, we deliver what we expect, and then we still try and see if there’s something else we can help them with. And we feel like that’s been a good way to approach this type of technology.
Jessica Ortiz, CCI:
Thanks for explaining that, Matt. I see how this is working. It’s been successful in some ways – it’s been helping patients get connected when they are forgetting their passwords, which I know is a huge challenge, I’d love to hear about what the biggest challenges have been so far in this project, and any advice you have for listeners in thinking about implementing a similar technology.
Matt White, CCHS:
Yeah. I mean, I think one of the challenges is always getting alignment with the sponsor or the requester operations. I mean, it’s easy to do the technology part, I mean even then it requires coordination, right? You’re working across teams; you’re working across systems. There’s often integrations and things that need to happen, but I always think the technology piece is relatively easy to figure out. It’s the coordination with the human side of things and change management that always kind of come into play. So as much as possible, we’ve tried to include our customers as part of the work. So, it’s like, “If you want us to do something for you, we want you to come to the meetings. We want you to participate in the solution design. We want you to be part of testing the solution, and we want you to get credit when things work.”
And so, I think that’s been a learning as we’ve gone through it, and some groups have been more willing to participate in the process, but we do feel like that’s really led to more timely solutions that deliver more value. So that’s been the approach that we’ve taken. And then I think it’s always, “How do you track what is working and isn’t working?” And just because something worked a couple of months ago doesn’t mean it’s going to continue to work. We often found that somebody would go in and make changes to the call tree –and messaging was different, and things got moved, and then we would see that – we would see a decrease in the outcomes that we were tracking. But if we weren’t doing that, we wouldn’t have known.
So, it’s, “How do you create the structures around the work that you’re doing so that you don’t revert back to the previous state and how you can continue the work and improve?” And then ultimately figuring out what your process measures are, what your outcome measures are, and to make sure that you’re delivering value with the work you’re doing. I make it sound like it’s not that hard, but it’s work to get these solutions deployed. And I think you’re doing yourself a disservice if you’re going out and you’re paying for a bunch of these services, and then not getting the value from them.
Jessica Ortiz, CCI:
Thanks for sharing about your challenges, Matt. I’m wondering, you were talking about how it’s really important to have some sort of structure to know what’s working well – what’s not. Do you have any advice, anything that you’ve tried in creating structure that maybe others can learn from?
Matt White, CCHS:
I mean, I will say that working with some earlier-stage startups, they usually want to be really helpful. And I think it’s important to make sure to pick things to work on that align with your priorities and their priorities, or at least you’re really clear on the expectations of why you’re doing something, and how long you’re going to be doing it for, and what you’re hoping to get out of it just so your paths kind of stay aligned. So that’s been a lesson that I’ve learned with various startups that I’ve worked with at this point.
And it’s also as you become a reference customer for them and they usually do a little bit more for you and, “How do you develop that kind of relationship that’s focused on the solution, and the problem space that they’re trying to solve?” And I think that has worked out well for us. And then I think internally, like I said, it’s setting some expectations and it’s making sure that you have the right people that have some skin in the game. And, it is helpful to have the priorities I guess, aligned internally with, “What is it we’re doing? Why are we doing it?” and have some data to support it.
They’re not cheap. I think healthcare, we’re too often focused on cost instead of this idea of value, and understanding the reasons why you’re doing something – and recognizing that the skillset of the work may change as you deploy these tools. And just thinking about, “How do you then support that?” – I think is another kind of important consideration for the work. It’s something that we’re still working through, but the idea of, “How do you transition this stuff from an idea to a solution to operations, so that you can actually do more of this?”And you don’t get slowed down, because of the good work that you’ve already done.
Jessica Ortiz, CCI:
When you’re talking about skills and training, as we talking about the workforce and as this changes as you implement these technologies, and it kind of shifts the different roles that people are playing –I think that’s a big topic of conversation where we’re thinking about AI in general, and what’s work going to look like in the future, and thinking about how can we best utilize people’s skills, and evolve those skills over time to meet the human needs, versus what some of these technologies can do. I’m curious, as we kind of wrap up this conversation, where you are now with this project, and is there any next steps, any next project involved with Hyro? What’s the status update at this point?
Matt White, CCHS:
Yeah. I mean, I think we’re always looking for new use-cases. I think we’d really like to start to take advantage of more of the natural language abilities. So, if somebody calls in, instead of saying, “I’m going to read you six options,” – it’s going to take a minute – we can just say, “Why are you calling?” And then we can match people to the appropriate response. It’s just a much more natural way of talking, and just want to be thoughtful that it provides the level of experience that we want, but I think that’s an area we’d like to do more of.
It also allows us to get more data and analytics and really understand those conversations, which we’re missing right now. We also want to enable more self-service options, especially for people that don’t have MyChart. So, if people want to schedule an appointment, change an appointment, cancel an appointment – how can we enable more of those things through voice, or text people a link when they call, and allows them to continue a conversation via a chat-assist. And then potentially escalate to a live agent depending on the conversation, or at least capture that conversation so the agent has that context. And I think we get that experience with Amazon, we get that experience with airlines now –banking… So, it’s like, “How do we start to deliver that in healthcare, and really make it easier?” And again, part of Hyro’s approach is once you have a solution, you can deploy it via various channels. So, you can enable it via text, via voice, via web, and so you can start to provide that consistent experience across different channels. So that’s some of the areas that we’re hoping to go next.
Jessica Ortiz, CCI:
That’s exciting, Matt. Well, really looking forward to continue following the work that you’re doing with Hyro. Of course, you are one of the most innovative organizations and team at Contra Costa in California, and I would also say across the country. So really looking forward to hearing more about the work that you are doing. And thank you for spending the time with us and sharing your experience with us today.
Matt White, CCHS:
Thank you so much for all you do and appreciate the time.
Health Pilots podcast outro:
Thanks for tuning into Health Pilots hosted by the Center for Care Innovations, with podcast production services by Wayfare Recording Company. If you like what you heard, please spread the word and be sure to subscribe where you listen to podcasts. For more information about the Center for Care Innovations or for upcoming programming, events, and funding opportunities, please visit our website – careinnovations.org – and connect with us on LinkedIn and Twitter. We’ll catch you here on the next episode of Health Pilots!
