Artificial intelligence chatbots are becoming more common in healthcare, but they come with serious risks. A new report highlights how these tools, while helpful, can also cause harm if not used carefully. Experts warn that relying too much on chatbots without proper oversight could lead to mistakes that affect patient safety.

The Rise of AI Chatbots in Medicine

AI chatbots like ChatGPT, Claude, and Gemini are designed to generate human-like responses to questions. They’re not regulated as medical devices and haven’t been validated for healthcare use. Still, many clinicians, patients, and health workers are turning to them for quick answers. More than 40 million people use ChatGPT daily for health info, according to OpenAI.

While these tools can offer useful guidance, they also have significant limitations. They generate answers based on patterns learned from training data, not actual understanding. This means they can confidently give false or misleading information, which may lead to patient harm if trusted blindly.

The Hidden Dangers of Relying on Chatbots

The report from ECRI, an independent patient safety organization, points out many safety concerns. For instance, chatbots have suggested incorrect diagnoses, recommended unnecessary tests, and even promoted subpar medical supplies. In some cases, they’ve invented medical terms or body parts, sounding like trusted experts but providing false information.

One dangerous example involved a chatbot advising that placing an electrosurgical return electrode over a patient’s shoulder blade was acceptable. That advice was wrong and risky, as it could have caused burns. Such errors show how AI tools can lead to serious medical errors if used without human oversight.

Experts also warn that as healthcare costs rise and clinics close, more patients might turn to chatbots instead of seeing a doctor. This could increase risks because chatbots aren’t equipped to handle complex medical issues or provide personalized care. Over time, this could worsen health disparities, especially if biases in training data influence the responses.

Chatbots tend to sound confident, even when they are wrong. They don’t truly understand medical context, which makes their advice unreliable. Medical professionals emphasize that AI can support healthcare but should never replace human judgment and expertise.

The Need for Caution and Oversight

According to the report, healthcare providers and patients need to be cautious when using AI chatbots. Proper guidelines, regulations, and oversight are essential to prevent harm. The goal should be to use AI as a tool to assist professionals, not as a substitute for their expertise.

For example, ECRI is hosting a live webcast on January 28 to discuss these hidden dangers. The session will focus on how to recognize and mitigate risks associated with AI in health settings. It’s a chance for stakeholders to learn more about the importance of disciplined oversight and understanding AI’s limits.

In conclusion, while AI chatbots hold promise for improving healthcare, they also pose significant safety challenges. Balancing innovation with caution is key to ensuring these tools enhance patient care without introducing new hazards. Healthcare systems must develop clear policies to guide the safe and effective use of AI in medicine.

Inspired by

• https://ai-techpark.com/misuse-of-ai-chatbots-tops-annual-list-of-health-technology-hazards/