ChatGPT and Gemini in warfarin counseling

Abstract

Aim To compare the accuracy, scientific adequacy, and clarity of responses provided by ChatGPT-4o and Gemini to frequently asked patients’ questions about warfarin use. Methods Forty patients’ questions were posed to ChatGPT and Gemini using the zero-shot method. Four cardiovascular surgeons evaluated the responses for adequacy, scientific accuracy, and clarity on a 5-point Likert scale. The first and second set of data evaluations were separated by seven days to minimize any memory effect. The experts were blinded to the type of LLM that created the response. Results ChatGPT responses were significantly shorter (187.3±47.6 vs 291.4±98.1 words; P<0.001) but scientifically more adequate (4.38±0.30 vs 4.17±0.35; P=0.004). Gemini scored higher in terms of clarity (4.69±0.24 vs 4.48±0.33; P<0.001). The two models did not significantly differ in terms of accuracy (P=0.606). Conclusion Both LLMs provide reliable and understandable information for warfarin counseling. While ChatGPT provides dense and scientifically detailed responses, Gemini excels in clarity and user-friendly communication. However, expert supervision and appropriate clinical guidance are critical for safe and comprehensive guidance.