Writing patient messages with AI: Time-saver or time drain?

Researchers are warning that physicians who use AI to create patient messages as a time-saver may end up spending more time cleaning up errors and removing extraneous information in those messages.

Clinicians are embracing AI as a time-saver for nonclinical tasks, and AI patient messages are one of the top items on the list. But researchers from Dartmouth College say that physicians may need to spend more time editing AI-generated patient messages than it would take to write them.

A study presented at the annual meeting of the Association for Computational Linguistics found that AI-generated answers were frequently too long and didn’t “align” with messages that physicians would write. Patient messages created with AI, for example, failed to ask follow-up questions and used irrelevant or inaccurate medical details.

Researchers offered the following example: When a 32-year-old woman taking an acid reflux drug complained she was experiencing constant nausea, AI suggested the patient might need to adjust her diet. A physician, by comparison, asked if there was any chance that the patient was pregnant.

Researcher Sarah Masud Preum, MD, warned that by missing out on clues that a physician would catch, AI could end up forcing physicians to devote too much cognitive energy to “playing AI janitor and fixing mistakes.”

The good news, however, was that tweaking AI-generated patient messages to reflect how individual physicians communicate improved message accuracy by 33%. The study also found that making those kinds of tweaks also reduced the time physicians spent on editing by 26%.

The researchers created a system to train AI platforms to better match physicians’ communication style when drafting patient messages. They found their system improved patient messages sent via six popular AI platforms, which could save physicians using the technology an hour or two of work a day.

Researchers also found that keeping AI-generated patient messages short can help physicians save time. That’s because it’s easier to make small edits to a shorter message.

“If you have to edit 75% of the message, you may be spending more time and energy on making changes than if you were to just write it from scratch,” said Timothy E. Burdick, MD, a member of the research team.

And finally, no matter how good AI gets at generating patient messages, Burdick said that physician oversight will always be necessary.

“I don’t foresee a time when the portal can respond to a patient without a clinician editing it first,” Burdick said. “But as we make the models better, we’ll be able to address portal messages much more quickly and with less mental energy.”

The study examined 146,000 conversations between 10,000 patients and their PCPs, comparing AI-generated replies to responses created by clinicians. The study evaluated physician responses drafted by AI platforms including Claude, Gemini and ChatGPT along with smaller commercial platforms like Llama, Aloe and Qwen.

A press release with more details on the study is on Dartmouth’s website.

The post Writing patient messages with AI: Time-saver or time drain? appeared first on Today's Hospitalist.