A recent study has raised concerns about the safety of ChatGPT Health, an AI-powered health service developed by OpenAI. The research warns that in many cases, the system fails to recommend emergency care when it’s actually needed. This has sparked discussions about the reliability of AI tools in critical health situations.

Study Highlights Risks in Emergency Scenarios

The study, published in Nature Medicine, tested ChatGPT Health using 60 realistic patient cases. These scenarios ranged from mild discomfort to serious medical emergencies. Three doctors evaluated each case beforehand to determine the appropriate level of care, and then compared their assessments with the AI’s recommendations.

The results showed that in over half of the cases where patients should have been advised to seek immediate hospital care, the AI recommended they stay home or see a regular doctor instead. While the system performed better in clear-cut emergencies like strokes or severe allergic reactions, it struggled with more complex or ambiguous symptoms.

Limitations and Safety Concerns

The researchers also pointed out issues with how ChatGPT Health handled sensitive situations like suicide risks. They found that the system’s warning functions sometimes disappeared depending on additional information added to the scenario, raising questions about its consistency and safety in mental health crises.

OpenAI responded by saying the study does not reflect typical use of the service. They also mentioned that the AI model is constantly being updated to improve its safety and accuracy. However, experts warn that these findings highlight the need for caution when deploying AI in healthcare, especially for urgent or life-threatening issues.

Overall, the study suggests that while AI tools like ChatGPT have potential, they are not yet reliable enough to replace professional medical advice, particularly in emergency situations. Users should be aware of these limitations and seek immediate care through traditional means if necessary.

Inspired by

• https://www.computerworld.com/article/4139545/researchers-warn-about-chat-gpts-new-health-service.html

Sources

• theguardian.com