AI is expected to become a transformative force, but applications in the medical field require 'extreme caution'

Las Vegas — While testing an AI-based reply drafting tool at the University of Illinois Hospital & Health Sciences System, a patient misspelled a medication name, the system's chief health information officer, Karl Kochendorfer, recalled during an HLTH conference panel discussion last week. The error caused the AI to provide side effects for a medication the patient was not taking, because the nurse did not carefully review the reply.

Ultimately, it did not cause a major problem — just a phone call to the patient or another message to correct it, he said. But it could have had serious implications for the tool.

"It almost killed the pilot... and it happened on day one," he said.

As the healthcare industry grapples with how to safely implement AI, investors and health systems are first seeing the potential of adopting automated administrative and back-office tools that could ease physician burnout and pose less risk to patient care, experts said at the HLTH conference.

But the pressure to adopt technology is high. Supporters argue AI could help address a major workforce challenge facing healthcare: by 2028, the U.S. will face a shortage of more than 100,000 critical healthcare workers as the overall population ages and requires more care, according to a report from consulting firm Mercer.

Although AI could be transformative, experts said the industry must proceed cautiously when implementing emerging tools. The stakes are high, as policymakers and experts have raised concerns about accuracy, bias, and safety.

Implementing AI in healthcare is complex, and the industry should learn from some predictive tools deployed previously, Rohan Ramakrishna, co-founder and chief medical officer of health information app Roon, said in an HLTH panel discussion.

"I think one thing we've learned is that applying AI solutions in a healthcare setting must be done with extreme caution," he said.

How AI could help address the 'simple mismatch between supply and demand'

AI could help alleviate one of the biggest problems in healthcare: more older patients with more complex conditions and more chronic diseases, but fewer doctors available to help, Daniel Yang, vice president of AI and emerging technologies at Kaiser Permanente, said in a panel discussion.

As older Americans need more care, millennials — the largest generation in the U.S. — want a more on-demand consumer experience, he added. But that is difficult to achieve with limited supply. Training a new doctor takes years, and fewer doctors mean more burnout, delayed care, and higher costs.

"This isn't even about AI; it's about what I think is a general problem in healthcare," Yang said. "What we're seeing is a simple mismatch between supply and demand."

AI can enhance clinician workflows, potentially helping them provide better care. Yang said an algorithm developed by researchers at The Permanente Medical Group saves about 500 lives annually by flagging patients at risk of clinical decompensation, meaning deterioration.

The technology could also reduce burnout and improve retention by cutting the time doctors spend on administrative tasks like notes. Providers have long reported spending hours on electronic health records, often to the detriment of patient care.

Christopher Wixon, a vascular surgeon at Savannah Vascular Institute, said he had come close to leaving medicine as the industry shifted to electronic health records. Gathering information while listening to patients was a challenge, and it was easy to miss nonverbal cues when forced to focus on a laptop screen.

But ambient documentation — where AI tools typically record conversations between clinicians and patients and draft notes — was a game changer, he said.

"It's better for me because it saves time," Wixon said in the panel discussion. "But at the end of the day, it's better for patients because they feel heard. It's been a truly transformative experience for me."

Investors and health systems focus on administrative burden

Given providers' heavy administrative workload and concerns about errors or bias in models involving clinical decisions, products that address administrative issues are among the top priorities for AI adoption.

Some investors also prefer automating these operational tasks.

"We will continue to focus on the 'unsexy' back-office automation to truly reduce the burden on the workforce, while setting aside clinical AI for now," said Payal Agrawal Divakaran, partner at .406 Ventures.

Administrative AI has also attracted more venture capital funding this year, according to a report released earlier this month by Silicon Valley Bank. Year-to-date in 2024, administrative AI companies raised $2.4 billion, compared to $1.8 billion for clinical AI, possibly due to lower regulatory and institutional barriers, especially for decision support tools.

"All the action we're seeing is on administrative tasks, also on prior authorization and low-value tasks," Megan Scheffel, head of credit solutions for life sciences and healthcare banking at Silicon Valley Bank, said in an interview. "You can move office staff to higher-value projects."

There are many opportunities now to use large language models as drafting tools, including for notes or nurse handoff documents, Greg Corrado, distinguished scientist at Google Research and head of health AI, said in a panel discussion.

Oversight is built in because providers must review the output before finalizing it. It is also easier to evaluate quality by asking about user experience or checking how many edits they need to make, he said.

But when evaluating operational or administrative tools, one must still be methodical, testing with the health system's local patient data, Todd Schwandt, partner at Cleveland Clinic Ventures, said in an interview. Cleveland Clinic's governance structure also focuses on issues such as the data used in the tool, how information is protected, and whether the product is safe and positively impacts patient care.

Administrative or operational products, such as ambient scribes or revenue cycle management tools, are a safer starting point.

"You're not taking the risk of making a clinical decision, right?" he said. "That level of trust doesn't exist yet. I think it will take time."

Preparing for AI deployment

Although AI may show promise in alleviating physician burnout, health systems face challenges in engaging doctors, establishing pilots, developing governance policies, and scaling products.

In one example shared at HLTH, St. Louis provider BJC decided to use vendor data to identify doctors who spent days signing off on documentation or wrote long notes, for an ambient note pilot, said Michelle Thomas, associate chief medical information officer and chief medical information officer for ambulatory at BJC Medical Group.

"I think only one out of 20 people responded. So we immediately thought, the ones who need it most aren't interested," she said in the panel discussion.

They decided to change direction and invite anyone interested to join the pilot — which quickly received responses, Thomas said.

Still, it is important to consider which providers should be involved in testing, because some doctors do not understand the requirements of participating in a pilot, such as needing to take time to send feedback or deal with frustrations with a new product.

Health systems should also consider what outcomes they want when adopting AI tools. Many doctors did not see much time savings because they spent a lot of time editing notes — not correcting errors — but making notes fit their own writing style, Thomas said. In contrast, advanced practice providers signed off on ambient notes more quickly after review.

"You really have to decide what your return on investment is. Are you looking for financial savings? Time savings? Hard numbers to justify the technology? Or something softer? Are you looking for patient satisfaction?" she said.

Cybersecurity — already a widespread challenge in healthcare — is also key to AI deployment.

Organizations should consider the same questions as with any system using protected health information, Melanie Fontes Rainer, director of the HHS Office for Civil Rights, said in an interview.

If they have a relationship with developers, is there a business associate agreement? Have they considered deletion policies for data stored in the cloud? Have they considered who needs access to the data?

"I think there's definitely a balance needed here, but it requires all of us to be responsible, to think about how we use this information, and how it could harm our systems, our patients, and how we take proactive measures to protect it," she said.