A new study published in the journal Nature Medicine raises concerns about the safety of Open AI’s health service ChatGPT Health, which in many cases fails to recommend emergency care when it ‘ actually needed, according to The Guardian.

Researchers tested ChatGPT Health with 60 realistic patient scenarios, ranging from mild discomfort to acute medical conditions. Three doctors assessed in advance the level of care required, and the results were then compared with the AI tool’s recommendations. In more than half of the cases where a patient should have been sent to the hospital immediately, the system instead advised them to stay home or get a regular doctor’s appointment.

According to the study, the service performed better in clear emergency situations, such as strokes or severe allergic reactions, but had trouble handling more complex or ambiguous symptoms. The researchers also point to shortcomings in how the system handled suicide risk, where warning functions sometimes disappeared depending on what additional information was added to the scenario.

In response, OpenAI said the study does not reflect how the service is typically used in practice and that the model is continuously updated, The Guardian reported.