What Does It Mean for LLMs to Be Empathetic_A Co-design Study of Patient Trust in LLM-Based Chronic Care for Heart Failure

Recent advances in Large Language Models (LLMs) offer transformative potential for chronic disease management. However, LLMs may exhibit “pseudo-empathy”, which can undermine patient trust and safety. This study investigates how patients define and perceive “meaningful empathy” when interacting with LLMs. This study uses University Health Network’s (UHN) Medly Chat, an LLM-powered chatbot for heart failure (HF) patients, as a use case to investigate patient expectations of LLM-generated empathy. We conducted a co-design session with 5 participants with lived experience of HF and informal caregivers. Participants scored the level of empathy on a 1–3 Likert scale across three clinical scenarios and shared rationales for their ratings. Quantitatively, most responses were rated as somewhat empathetic (67%), with fewer rated as empathetic (20%) or not empathetic (13%). Qualitative findings indicate that overly human-like expressions (e.g., “I hear you”) were frequently perceived as condescending and associated with lower empathy scores. Participants preferred LLMs to display empathy by acknowledging concerns and providing actionable guidance in a neutral, information-focused tone. These findings suggest that, while the LLM maintains a baseline level of perceived empathy, mimicking human empathy is not necessarily the gold standard in LLM-supported clinical interactions. Developers of LLM-powered patient-facing chatbots should consider prioritizing acknowledgment and actionable guidance and evaluate LLM performance accordingly. Future work involves refining the Medly Chat prototype to reflect patient expectations and priorities.