ChatGPT has reportedly failed to clear yet another top exam.
A study in the journal Urology Practice revealed that the AI chatbot developed by OpenAI fared poorly, achieving less than 30 per cent in the American Urologist Association's Self-Assessment Study Program for Urology (SASP).
In the report, Christopher M Deibert from the University of Nebraska Medical Center, said that ChahGPT not only has a low rate of correct answers regarding clinical questions in urological practice but also “makes certain types of errors that pose a risk of spreading medical misinformation”.
As per the report, the accuracy percentage was specifically 26.7% for open-ended questions and 28.2% for multiple choice questions.
In addition, the chatbot occasionally provided “intermediate” responses which suggested a lack of precise or lucid answers.
It’s interesting to note that accuracy further dropped when the Large Language Model (LLM) tried to recreate its solutions, suggesting that this particular study programme may have challenges in generating correct and trustworthy responses.
As stated in the report, the majority of ChatGPT’s answers to open-ended questions in the SAPS concentrated on elaborating on the chosen response. Although the replies produced by ChatGPT were longer than those offered by SAPS, the research said that they were “frequently redundant and cyclical in nature.”
Dr Deibert said, “Overall, ChatGPT often gave vague justifications with broad statements and rarely commented on specifics.”
“ChatGPT continuously reiterated the original explanation despite it being inaccurate,” the report added.