Patients Not Told AI Drafted Messages From Their Doctors

Every day, physicians receive hundreds of thousands of messages sent by their patients through the communications features built into the MyChart patient records platform marketed by Epic Systems. Madison, Wisconsin-based Epic controls about 29 percent of the market for electronic health record (EHR) platforms in acute care hospitals, and by 2021, its systems held medical records for 250 million Americans—or about 75 percent of the population.

However, increasingly doctors aren’t writing the first drafts of replies to those messages. At more than 150 health systems and medical offices across America, a new generative artificial intelligence function in MyChart now writes over a million of those reply drafts every month.

Many of the patients who receive those replies have no clue that AI software originally wrote those messages, or wrote portions of the text those patients read along with edits by their doctors. Believe it or not, that’s because senior officials at several of the health systems who are using this new MyChart function told the New York Times that they don’t let patients know their replies contain content written by artificial intelligence. Those systems include New York University’s Langone Health System in New York City, the University of Wisconsin Hospitals and Clinics in Madison, Wisconsin, and Stanford Health Care in Stanford, California.

Several authorities have expressed alarm. They’re concerned because MyChart’s AI functionality doesn’t always write responses that are correct. Some experts have found potentially dangerous errors in such AI-drafted messages for patients, and they fear that busy doctors under time pressure might not catch and rewrite such incorrect passages.

In a June 2023 article here on MHAOnline.com titled “ChatGPT: Relieving Healthcare’s Administrative Burden,” we pointed out that healthcare organizations were mostly using artificial intelligence platforms outside of treatment. Typically these applications helped save time and effort in performing managerial and administrative tasks like appealing insurance claim denials.

That piece quoted Dr. John Halamka, the president of the Mayo Clinic Platform who explained how AI was useful “for generating text that humans then edit to correct the facts. The result is reduction of human burden. And if we look at some of the—what I’ll call the “crisis of staffing,” and the “great resignation” and retirement of our clinicians—burden reduction is actually a huge win.”

Also, on our partner platform OnlineEducation.com in October 2024 we quoted Babson College management professor Peter Cohan in our article titled “Is Education the AI Industry’s First Killer Application?” He says that most of the $150 billion in artificial intelligence investments during 2024 has not yet been deployed because companies are “terrified of getting sued” over increasing reports of AI mistakes and hallucinations. One such widely-reported verdict held Air Canada liable in February 2024 for negligent misrepresentation when its customer service chatbot cut a deal to refund a passenger’s ticket for a flight he never took.

But now critics fear that the widespread adoption of MyChart’s controversial artificial intelligence message-drafting feature has enabled AI to influence doctor-patient relationships and clinical decisions. As if communicating with patients wasn’t a meaningful part of clinical practice, this promotional video from Epic Systems shows how the company sells the message-drafting feature as a way for doctors to save time they can then devote to “more meaningful activities.”

It is true that clinicians are now facing increasingly heavy workloads required to write responses to patient email messages, a trend that developed because of the pandemic early in 2020 when office visits for routine diagnosis and treatment became unavailable from many overwhelmed practices. Measured by one study at 157 percent of the pre-pandemic average level, that torrent of email messages has never let up.

For example, a 2024 study published in the Annals of Family Medicine showed that family care physicians at the University of Wisconsin’s UW Health System were now spending an average of 10 hours every month writing email replies to patients—time that previously hadn’t been considered billable by the healthcare industry. That email avalanche resulted in recent decisions by several large health systems to charge patients up to $98.00 for each email reply, after allegations surfaced that some patients were relying on email conversations to avoid office visits.

Nevertheless, an April 2024 study published in the Lancet by researchers from the medical schools at Harvard, Yale, and Wisconsin found that the use of artificial intelligence large language models (LLMs) to write first drafts of email replies posed significant patient safety risks. They discovered that seven percent of 156 drafts—if left unedited—posed a risk of severe harm, including in one case a risk of death. Most of the harmful responses resulted when the AI platform incorrectly determined or conveyed the scenario’s acuity and action recommended to the patient.

The Lancet team also demonstrated that studies like theirs were insufficient to understand clinical risks and utility because of the threat of automation bias. Dr. Ken Holstein at Carnegie Mellon University’s Human-Computer Interaction Institute told the Times that this is a well-documented tendency for professionals to accept an algorithm’s recommendations–even if they contradict that professional’s expertise.

Because a clinician ideally is always reading through the AI-written drafts and correcting errors, healthcare administrators have often spoken of Epic’s AI message-drafting system as a “low-risk AI” application. However, Dr. Holstein told the Times that characterization “goes against about 50 years of research.” He says that because of automation bias, doctors could be less critical when reading through drafts written by the AI system, and potentially allow dangerous errors to slip through to patients.

In the Lancet’s study, the team emphasizes these potential risks in exhaustive detail:

We also showed that existing evaluations are insufficient to understand clinical utility and risks because LLMs might unexpectedly alter clinical decision making, and that physicians might use LLMs’ assessments instead of using LLM responses to facilitate the communication of their own assessments.

LLMs might affect clinical decision making in ways that need to be monitored and mitigated when used in a human and machine collaborative framework. The content of physician responses changed when using LLM assistance, suggesting an automation bias and anchoring, which could have a downstream effect on patient outcomes.

The improved inter-physician agreement and similarity of response content between LLM drafts and LLM-assisted responses suggest that physicians might not simply use LLMs to better phrase their own assessment, but instead adopt the assessment by the LLM. This finding raises the question of the extent to which LLM assistance is decision support versus LLM-based decision making. Additionally, a minority of LLM drafts, if left unedited, could lead to severe harm or death. Thus, there is a need for new approaches for evaluation and monitoring, especially as trust in LLMs builds and clinicians become less vigilant and more reliant on LLMs.

Released only a few months after the Times’ report, a fascinating March 2025 study suggests that a curious contradiction may exist among the opinions of patients about AI-drafted clinician messages. Conducted at Duke University and published by the Journal of the American Medical Association under the title “Ethics in Patient Preferences for Artificial Intelligence–Drafted Responses to Electronic Messages,” this survey investigated preferences among Duke University Health System patients for AI-written clinical replies.

The findings revealed that respondents expressed a slight preference for clinician messages drafted by artificial intelligence over those drafted by a human. Nevertheless, that was true even though those respondents expressed higher satisfaction ratings with messages that they were told were drafted by their clinician—or when there was no disclosure—over those they were told had been drafted by AI.

The AI-generated messages were longer and more detailed, and patients ranked them higher in usefulness and empathy. Interestingly, the patients’ satisfaction ratings slightly decreased when they learned that AI wrote a message. Nevertheless, overall satisfaction remained high, with more than 75 percent of patients reporting satisfaction with the messages they received.

In their discussion of the findings, the authors offered an interesting perspective on this preference/satisfaction contradiction’s significance:

This contradiction is particularly important in the context of research showing that increased access to clinicians via electronic communication improves patient satisfaction, while evidence linking the in-basket to burnout is prompting development and use of automated tools for clinicians to reduce time spent in electronic communication. The lack of difference in preferences between human vs no disclosure may indicate that surveyed participants assume a human author unless explicitly told otherwise.

 
The researchers also pointed out that the results suggest that unlike professionals, patients don’t experience automation bias. In fact, the researchers believe that their findings might demonstrate a reverse automation bias among patients, who prefer those messages that they believe come from their clinicians rather than messages from an artificial intelligence program.

However, it remains unclear whether these results will generalize to the population because the sample doesn’t appear to represent a cross section of most Americans. Duke’s 1,455 survey respondents had a median age of 57 years, were 63 percent white and 63 percent female, and a whopping 81 percent hold college or graduate degrees.

California’s legislature grew so concerned that patients might not know that an AI program had drafted their doctors’ messages that it decided to regulate these communications in legislation known as AB 3030. After several medical industry lobbying groups withdrew their objections, the Assembly unanimously approved this bill, and Governor Gavin Newsom signed it into law in September 2024. The measure took effect in California on January 1, 2025.

MHA students and alumni would be wise to familiarize themselves with AB 3030 because many states are likely to enact some form of this legislation within the next few months. It’s also not a surprise that this law appeared first in the bellwether state of California, home to Silicon Valley and some of the biggest names in the AI industry like OpenAI, Perplexity, Anthropic, and Google.

This bill mandates significant disclosure requirements on any facility, clinic or practice that uses generative AI for clinical communications with patients. The law consists of two main provisions:

1. Disclaimer

A disclaimer has to inform the patient that an artificial intelligence platform generated the communication. In email messages and other written communications, that disclaimer has to appear at the communication’s outset, and must remain throughout any continuous interaction that takes place online. A similar version of this disclaimer requirement also applies to audio and video.

2. Clear Instructions

The communication must provide clear instructions about how a patient can contact an individual who works for the clinic or practice.

Only a single exception exists, and that is available any time the AI-drafted message has been “vetted,” which means the AI’s language was reviewed by a licensed or certified healthcare provider. But with only one exception, this law may discourage practices from adopting artificial intelligence-based drafting functions in platforms like MyChart before the practice has systems and procedures in place to provide this vetting.

Practices also need to follow these procedures consistently. These strict disclaimer and instruction requirements apply to every communication with the patient based on AI-written language, even if the patient already knows or should know that the practice routinely uses AI to draft their communications with patients.