Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
Advertisement
Scientific Reports volume 16, Article number: 16070 (2026)
4245
4
Metrics details
Large language model (LLM)-based chatbots have been utilized across various healthcare domains and have garnered substantial attention. This study aimed to evaluate and compare the performance of several LLM-based chatbots with that of medical students in responding to neurology questions. This cross-sectional study, conducted in December 2025 in Iran. ChatGPT-5, Gemini 3, Copilot 2025, Perplexity, and 20 medical students responded to a neurology questionnaire. A confusion matrix was utilized to analyze the data. In this regard, four metrics—sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV)—as well as overall accuracy were calculated. Moreover, correlations examined chatbot performance against question characteristics (word count, context, format, type, modality, language). The study revealed that overall performance metrics for the evaluated chatbots significantly outperformed those of medical students (p < 0.001). Among the evaluated chatbots, Copilot exhibited superior performance (0.88), followed by ChatGPT-5 (0.86), in terms of accuracy. Meanwhile, quantitative question types were associated with a significant reduction in chatbot performance (r = 0.470, p = 0.001). The study findings presented valuable insights results particularly pertinent to neurology, where chatbots can serve as supplementary tools for practitioners, enhancing diagnostic accuracy and clinical decision-making while adhering to established ethical standards. However, further research is required to provide more precise insights, particularly with a larger sample size of human participants.
Artificial Intelligence (AI) generally refers to the use of computers and technology to simulate intelligent behaviors and carry out processes that involve critical thinking. More specifically, AI is a branch of science and engineering concerned with designing and developing machines or systems capable of reproducing human cognitive functions such as learning, reasoning, perception, problem-solving, and decision-making1. In such context, Large Language Models (LLMs) are deep learning models trained on vast amounts of textual data, enabling them to generate natural, human-like language. These models are employed across various fields, including medicine and the sciences, for data analysis and content generation, and they possess significant transformative potential2,3. In this context, several LLMs have also been developed in the form of chatbots, while conventional chatbots employ such models to analyze data and generate responses to user queries conveyed through conversational interactions4,5.
The evidence indicates that obtaining information from web-based platforms is of considerable interest to users within the healthcare context6,7,8. The evidence suggests that the information provided should not only be of sufficient quality in terms of accuracy, but should also be readable and understandable for the vast majority of users with diverse literacy levels9,10,11,12. Early literature claims that chatbots are more appropriate for informational aspects of healthcare services than for domains requiring clinical interventions or direct patient care13. Meanwhile, the deployment of chatbots—and AI in general—within healthcare settings has engendered several ethical concerns that necessitate rigorous examination14,15. However, the literature also indicates that ChatGPT demonstrates moderate to good accuracy on medical specialty examinations and clinical questions. For example, this chatbot has achieved strong scores on neurology board examination questions16.
Neurological services play a pivotal role in mitigating the foremost contributors to global disability and premature mortality, especially amid the escalating prevalence of neurological disorders in aging populations17. Conditions including dementia, Parkinson’s disease, epilepsy, and stroke constitute the predominant drivers of morbidity and disability in certain regions, imposing substantial burdens on healthcare infrastructures due to heightened requirements for diagnostics, therapy, and rehabilitation17,18.
Conducting a bibliometric analysis of the existing literature, incorporating the most recent data on the subject, is highly valuable for achieving a comprehensive understanding of the overall research landscape. This approach is particularly important when undertaking new studies, as it enables researchers to utilize data from prior publications to generate novel insights into the topic19. In such context, a 2023 study evaluated ChatGPT’s performance on 509 multiple-choice neurology board exam questions (excluding image-based items), assessing initial attempts and up to three consecutive tries. It achieved 65.8% accuracy on the first try (26th percentile vs. humans) and 75.3% after three (50th percentile), excelling in pain (100%), epilepsy/seizures (85%), and genetics (82%), but underperforming in imaging/diagnosis (27%), critical care (41%), and cranial nerves (48%). The results demonstrated ChatGPT’s capacity to match neurology residents on specialized queries, with accuracy improving across attempts, underscoring its potential in medical information processing and education16. A 2024 study compared ChatGPT’s performance against 120 neurologists taking a 2022 specialist exam in Valencia, Spain. ChatGPT-3.5 correctly answered 54.5% of questions, ranking 116th out of 122 and failing, whereas ChatGPT-4 achieved 81.8% accuracy (score: 7.57/10), ranking 17th and outperforming many specialists, with superior reliability. The results indicated that advanced ChatGPT versions can match or exceed human specialists in specialized medical knowledge assessment, holding substantial potential for medical education and evaluation20. Yet another study published in 2025, assessed the accuracy of AI platforms—including ChatGPT-4.0, GPT-4o, Gemini 1.5 Pro, Claude 3.5, Perplexity, and Perplexity Pro—in diagnosing, treating, and managing multiple sclerosis (MS) using 20 expert-designed multiple-choice questions posed to 37 Turkish neurologists and 79 neurology residents. Specialists averaged 12.05 correct answers (MS clinic specialists: 17.67), residents 9.08 (higher with > 2 years’ experience), while AIs averaged 17 (Claude 3.5 highest at 19). The findings suggested AI’s supportive role in MS care, though challenges persisted in complex cases, warranting further research21. Overall, the literature’s lack of comprehensiveness—such as the omission of image-based modalities in earlier studies—and the recurrent calls for further investigation in recent research underscore the critical need for additional studies in this domain.
Assessing and comparing the performance of AI LLM-based chatbots and medical students in answering neurology-related questions, using a comprehensive questionnaire that includes items with varying characteristics—such as length and word count in both the questions and answer options, their format (descriptive or analytical), their type (quantitative or qualitative), their modality (textual or visual), and their language—would contribute valuable and novel data to the existing literature. Such research offers several benefits for stakeholders. Policymakers, health system administrators, and neurology service providers can use these findings to gain a general understanding of the capabilities of AI—particularly its accuracy and reliability in neurology—and to incorporate this evidence into planning for the integration and application of LLM-based chatbots within healthcare delivery processes, especially in this specialty. Developers and designers of LLM-based chatbots can also leverage the insights derived from such research to identify the strengths and weaknesses of their systems in responding to clinical questions and subsequently take steps to enhance their performance.
The initial evaluation of the study chatbots’ performance in responding to the questionnaire was conducted using TPs, TNs, FPs, and FNs. The results indicated that ChatGPT-5 achieved 37 TPs, Gemini achieved 36 TPs, Copilot achieved 40 TPs, and Perplexity achieved 31 TPs. With respect to true negatives, ChatGPT-5 recorded 6 TNs, Gemini 5 TNs, Copilot 4 TNs, and Perplexity 5 TNs. In terms of false positives, ChatGPT-5 produced 5 FPs, Gemini 9 FPs, Copilot 6 FPs, and Perplexity 11 FPs. Finally, regarding false negatives, ChatGPT-5 recorded 2 FNs, while Gemini and Copilot recorded no false negatives, and Perplexity recorded 3 FNs (Fig. 1). On the other hand, the initial evaluation of the medical students’ performance on the study questionnaire revealed mean values of 29.8 TPs, 3.35 TNs, 16.15 FPs, and 0.7 FNs, with corresponding standard deviations of 7.02, 1.87, 7.97, and 0.97, respectively.
Performance metrics of study chatbots.
The results of the study also presented the performance metrics of medical students and four chatbots—ChatGPT-5, Gemini 3, Copilot, and Perplexity—assessed using PPV, NPV, sensitivity, specificity, and overall accuracy, as described earlier. The overall performance metrics of the study participants and the chatbots are presented below. Figure 2 presents the comparison of medical students and chatbots in terms of accuracy.
Accuracy comparison between medical students and chatbots.
Medical students exhibited a PPV of 0.65 and an NPV of 0.85. Sensitivity was high at 0.97, whereas specificity was comparatively low at 0.20. The overall diagnostic accuracy among medical students was 0.66.
ChatGPT-5 demonstrated a PPV of 0.88 and an NPV of 0.75, with a sensitivity of 0.94 and a specificity of 0.54. The overall accuracy for ChatGPT-5 was 0.86.
Gemini achieved a PPV of 0.80 and an NPV of 1.00. Sensitivity reached 1.00, while specificity was 0.35. The overall accuracy for Gemini was 0.82.
Copilot showed a PPV of 0.86 and an NPV of 1.00, with a sensitivity of 1.00 and a specificity of 0.40. The overall accuracy recorded for Copilot was 0.88.
Perplexity recorded a PPV of 0.73 and an NPV of 0.62. Sensitivity was 0.91, and specificity was 0.31, resulting in an overall accuracy of 0.72.
Table 1 presents the findings of a paired-samples t-test conducted to compare questionnaire responses between chatbots and medical students. The table reports the mean paired difference along with its standard deviation, standard error, and the corresponding 95% confidence interval. In addition, the t value, degrees of freedom, and two-tailed significance level are provided. The analysis is based on 50 paired observations, as reflected by the reported degrees of freedom. In such context, as indicated by the table, the mean difference between chatbot and medical student performance was − 0.157, reflecting higher performance scores for chatbots. The paired-samples t-test revealed a statistically significant difference between the two groups, t(49) = − 4.142, p < 0.001, indicating that the observed difference is unlikely to be attributable to chance. Moreover, the 95% confidence interval for the mean difference ranged from approximately − 0.233 to − 0.081, with both limits below zero, demonstrating a consistent performance difference between chatbots and medical students at the population level. Collectively, these findings indicated the statistically significant superiority of chatbots over medical students in answering the study questionnaire.
As described in the methodology section, it should be noted that the data were coded such that lower values corresponded to correct responses (1 = correct) and higher values corresponded to incorrect responses (2 = incorrect). Accordingly, positive correlations reflected poorer performance of the chatbots, whereas negative correlations reflected better performance. In such context, as presented in Table 2, several moderate, statistically significant correlations with question type were observed, including ChatGPT-5 (r = 0.306, p = 0.031), Copilot (r = 0.345, p = 0.014), Perplexity (r = 0.309, p = 0.029), and the combined chatbot score (r = 0.470, p = 0.001), with the latter representing the strongest association. These findings indicated that chatbots tended to succeed or fail on the same types of questions (qualitative or quantitative), suggesting that quantitative question types were systematically challenging for the chatbots.
As indicated by the study findings, the overall performance metrics of the study chatbots significantly exceeded those of the medical students (p < 0.001), providing important insight into the potential application of chatbots in medical services. This finding is particularly relevant to the field of neurology, where chatbots may serve as supplementary tools alongside medical practitioners, with the potential to enhance the quality and precision of diagnostic processes and clinical decision-making within this context. In this regard, the existing evidence indicates the substantial potential of AI to be applied within healthcare decision-making domains, provided that its use adheres to established ethical standards22,23,24,25. These include employing AI as a supplementary tool rather than a replacement for human professionals, maintaining appropriate human oversight, and upholding key ethical principles such as privacy protection, bias mitigation, transparency, accountability, accessibility, fairness, and the clinical utility of outcomes14,15,26.
The findings of our study are consistent with those of a recent scoping review that examined the extant evidence on LLM-based chatbots for treatment decision-making in neurology. Although the review highlighted the potential of these chatbots to augment clinical care, it also underscored the current insufficiency of evidence to substantiate improvements in patient outcomes27. In this context, our study’s results offered valuable insights, lending support to the assertion that such chatbots yield positive effects on clinical decision-making within neurology.
As revealed by the study findings, chatbots demonstrated overall superior performance compared to medical students in providing correct answers. Specifically, ChatGPT-5 recorded 43 correct answers (86%), Gemini 3 recorded 41 correct answers (82%), Copilot recorded 44 correct answers (88%), and Perplexity recorded 36 correct answers (72%). By comparison, medical students achieved an average of approximately 33 correct answers (66%). These results underscored the chatbots’ superior accuracy relative to medical students. Notably, only one medical student (5% of the study participants) matched ChatGPT-5’s performance, which still remained below Copilot’s highest score among all evaluated chatbots. These findings aligned with prior literature demonstrating the superior performance of chatbots compared to medical students20,28,29,30,31,32,33. However, the superiority narrated by the literature had not been statistically significant and had been, at times, fragile, showing lower performance than humans, particularly in earlier studies that involved less capable versions of chatbots16,34,35.
The literature has also reported the apparently superior performance of Copilot within the given context which is in line with our study findings30. Moreover, the lower performance of Gemini observed in our study, when compared with other chatbots, was likewise documented in the existing literature35. However, in contrast to our findings—which indicated superior performance of Gemini relative to medical students—the literature had reported that Gemini demonstrates lower performance than humans, particularly those with specialized expertise within the given context35.
According to the study findings, sensitivity was high for both medical students and the chatbot systems, reflecting a strong capacity to correctly identify answers presented within the question options. In this regard, medical students demonstrated a sensitivity of 0.97. Meanwhile, among the chatbot systems, Gemini 3 and Copilot achieved perfect sensitivity values of 1.00, while ChatGPT-5 and Perplexity recorded sensitivities of 0.94 and 0.91, respectively. These results indicated effective TP detection across both human and chatbots. Nevertheless, Gemini and Copilot demonstrated the highest degree of sensitivity among all. In this regard, no neurology-specific literature reported LLM sensitivity values, preventing contextual comparison of our findings; nonetheless, our chatbot superiority in sensitivity corroborated evidence from other contexts36,37. Literature also corroborated Copilot’s superior performance in sensitivity relative to other LLMs37,38.
The study findings further indicated that specificity exhibited the most pronounced variation between medical students and the study chatbots. Medical students showed low specificity at 0.20, suggesting limited ability to correctly identify the “none” option as the answer. In contrast, the chatbots demonstrated higher specificity values, ranging from 0.31 for Perplexity to 0.54 for ChatGPT-5. Although specificity among the chatbots remained moderate overall, it exceeded that observed among medical students, indicating improved identification of TN cases. The existing literature within the field of neurology does not provide any data regarding specificity. However, literature from other domains corroborates our findings regarding Perplexity’s inferior performance regarding specificity relative to other chatbots39.
Overall accuracy results also underscored differences in performance. Medical students achieved an accuracy of 0.66, while chatbot accuracy values ranged from 0.72 for Perplexity to 0.88 for Copilot. In this regard, Copilot demonstrated the highest overall accuracy, followed by ChatGPT-5 at 0.86 and Gemini 3 at 0.82. The literature corroborated our findings of the superior accuracy of chatbots compared to professionals in the field of neurology21,32. However, chatbots have not consistently outperformed neurology health professionals in terms of diagnostic accuracy, particularly in studies requiring them to generate diagnoses without predefined response options and relying exclusively on their inherent knowledge base35. Although such findings also indicated that Gemini again exhibited seemingly lower performance compared to other chatbots—particularly LLMs—as reported by the findings of our study35. In this regard, newer versions of chatbots, particularly ChatGPT, have demonstrated superior accuracy compared to their older counterparts20,32,40,41.
The analysis of chatbot performance indicated that Q37 was the most challenging, with all four chatbots having produced incorrect responses. A detailed examination of the questionnaire item’s characteristics revealed that it was a question with a TN answer. Its primary context concerned the dosing principles of intravenous thrombolytic therapy in acute ischemic stroke. The item comprised 59 words, was both analytical and quantitative in nature, consisted solely of text, and incorporated words in both Persian and English languages. As the word count of this item was close to the mean word count of the questionnaire items (50.16, SD = 26.12)), it was not considered a contributing factor to the universal failure of chatbots to answer correctly. Rather, the failure could be attributed to the item’s quantitative and analytical nature, as well as its mixed Persian-English language composition. In such context, as indicated by our study findings, quantitative questions significantly lowered the performance of chatbots (r = 0.470, p = 0.001). In this regard, supporting evidence indicates that chatbots exhibit lower performance on analytical questions which needs critical thinking, particularly quantitative ones42,43,44,45. The contextual characteristics of the item did not appear to contribute to the chatbots’ suboptimal performance on this task, as evidence indicates their superior capability in addressing dosing calculations46.
Q2 and Q21 were also highly problematic, each having received three incorrect answers. Examination of these two questionnaire items revealed that both had TN answers. They comprised 68 and 96 words, respectively. The first addressed the clinical presentation, imaging patterns, and acute management of cerebral venous thrombosis involving deep cerebral veins, while the second concerned indications for initiating antiepileptic drug therapy following seizure events. Both were analytical in nature; one was qualitative and consisted solely of text in mixed Persian-English, whereas the other was quantitative and similarly consisted solely of text in mixed Persian-English. In this regard, previous literature has documented limitations of chatbots in imaging, epilepsy, and seizure-related contexts, resulting in reduced performance in these domains16,32. The evidently higher word count of the latter item might have contributed to the chatbots’ suboptimal performance, as existing evidence indicates a decline in their efficacy with inputs containing greater numbers of words47. Moreover, a detailed examination of the study results indicates that 50% of the questionnaire items with TN answers were not successfully answered by at least two chatbots. In this regard, although the literature suggests that chatbots exhibit limitations in specificity and the correct identification of TN options relative to humans, our results contradict this assertion37. Specifically, as presented by our study findings, all of the chatbots evaluated in this study outperformed humans in terms of specificity.
A limitation of the study was the lack of access to paid versions of the chatbots due to lack of resources. Moreover, the medical students in the study sample were drawn from a single university situated in a low-resource region, which may account for their comparatively lower performance. The number of medical students included in the study was limited to 20 due to constraints related to resources and participant accessibility. Additionally, differences in testing conditions between medical students and chatbots may be considered a study limitation, despite the implementation of several measures designed to ensure comparable testing conditions for both human participants and chatbots. Furthermore, the analytical approach employed in this study required the data to be analyzed in a manner that treated the inclusion of a “none of the above” option as the true negative category. This methodological choice may have influenced the interpretation of specificity and should therefore be explicitly acknowledged and justified as a study limitation. Future researchers are encouraged to address these limitations. The readability of chatbot-generated content could also be evaluated in future studies to yield more comprehensive insights into chatbot performance within this context.
The study yielded several important implications for its beneficiaries. Specifically, it provided valuable insights into the comparative performance of chatbots, both against one another and in relation to medical students. The findings demonstrated that chatbots significantly outperformed students in responding to the questions. Furthermore, the study highlighted the notable impact of quantitative questions, which were associated with lower chatbot performance. In addition, the research offered meaningful evidence regarding the sensitivity and specificity of chatbots. These insights are particularly relevant for healthcare policymakers, service providers, and—most critically—manufacturers of chatbots. Policymakers and providers can utilize the data to support the evidence-based integration of chatbots into healthcare services, while manufacturers may refine chatbot performance by systematically acknowledging and addressing their strengths and limitations. Finally, the study underscored the need for future research to build upon these findings, thereby generating deeper insights into the role and optimization of chatbots within healthcare systems.
The study demonstrated that the overall performance metrics of the evaluated chatbots significantly surpassed those of the medical students (p < 0.001). Among the chatbots, Copilot exhibited the highest performance, with ChatGPT-5 ranking second in terms of accuracy. Meanwhile, quantitative question types were associated with a significant decline in chatbot performance (r = 0.470, p = 0.001). These findings hold particular relevance for neurology, where chatbots may function as supplementary tools to medical practitioners, potentially improving the quality and precision of diagnostic processes and clinical decision-making, provided ethical guidelines for their use are strictly observed. However, future studies are necessary to offer more precise insights, particularly with a larger sample size of human participants.
This was an observational cross-sectional study conducted in December 2025 within the neurology department of a public hospital in Birjand, South Khorasan Province, Iran. Data from this study were reported in accordance with the CONSORT guidelines for cross-sectional studies48. The study was conducted in two phases, utilizing several chatbots as well as medical students. During the study, a number of freely available chatbots and a group of medical students were employed to answer research questions in the field of neurology. Subsequently, after aggregating the responses and evaluating their accuracy, the performance of the chatbots in answering the research questions was assessed using confusion matrix analysis to determine their precision and overall accuracy.
The primary objective of the study was to evaluate and compare the performance of several LLM-based chatbots with medical students in responding to neurology questions. Accordingly, two principal research questions were formulated as follows:
What is the performance of chatbots in answering neurology questions?
What is the performance of medical students in answering neurology questions?
Chatbots
The chatbots examined in this study included ChatGPT-5, Gemini 3, Copilot 2025, and Perplexity. These chatbots were selected due to their free accessibility on the global internet, their international prominence—particularly in the existing research literature—and their support for the Persian language.
Medical Students
In accordance with the literature, the number of medical students was set at 20 participants, as this was considered sufficient for conducting such research34. The inclusion and exclusion criteria for the selected sample were as follows:
Inclusion Criteria:
Enrollment in a medical degree program.
Completion of coursework related to neurology.
Exclusion Criteria:
Inability to respond to the questions in person and physically.
Approximately 60% of the medical students who participated in the study were female, with the remaining participants being male. The mean age of the participants was 24.7 years. Nearly 70% of the participants had passed at least 6 years of education and were in the final years of their program, while the remaining participants were trainees in the initial or middle years of their university education (Table 3).
Prior to data collection, the research questionnaire was developed by a panel of experts in the field of neurology to ensure the validity of the study results (Appendix 1. Research Questionnaire). The panel consisted of several university professors. The definitive answers to each item of the questionnaire were used as the reference for evaluating the accuracy of the responses provided by the chatbots and the medical students. Finally, these gold-standard answers were supervised and validated by another three specialists in the field of neurology to ensure their correctness and avoidance of bias.
In the next step, the research questions, consisting of 50 multiple-choice questions mainly in Persian, were administered to a sample of chatbots as well as medical students. The evidence presented in the literature proposed that this number of questions was sufficient34. In this regard, some of the words in certain questionnaire items were in English. The questionnaire included questions with varying characteristics in terms of length and number of words in both the questions and the answer options; their format, whether descriptive or analytical; their type, whether quantitative or qualitative; their modality, whether textual or visual, and language.
As presented in Table 4, the questionnaire primarily focused on clinical neurology and neurological emergencies. Its contexts included the diagnosis and emergency management of neurological disorders, specialized clinical neurology, sensory-motor and cranial nerve disorders, pharmacologic and emergency interventions, neuroimaging interpretation, and clinical examination and syndromes. The clinical topics of the questionnaire items encompassed vascular and stroke disorders, chronic central nervous system disorders, seizures and status epilepticus, headache and pain syndromes, vestibular and balance disorders, and neurological examination and localization. The structure and style of the questions incorporated clinical scenario-based items, imaging interpretation questions, and pharmacology and dosing questions. In this regard, consistent with the literature reporting the comparatively lower performance of chatbots on complex questions, some items were deliberately designed to be complex or scenario-based, requiring careful reasoning rather than rote memorization32,42. Overall, the mean word count of the questionnaire items was 50.16 (SD = 26.12). The distribution of analytic questionnaire items accounted for approximately 60%. Of the questionnaire items, 92% were qualitative in nature, 96% consisted solely of text, and 56% were presented in a mixed Persian-English format. The wide distribution of word counts among the questionnaire items enabled a comprehensive analysis of chatbot performance across varying lengths. Although the majority of items (92%) were qualitative, the inclusion of quantitative items afforded an opportunity to compare chatbot performance on quantitative versus exclusively qualitative content. Similarly, presenting items in multiple languages (56% in mixed Persian-English format), as opposed to a single language, allowed for an evaluation of chatbot performance relative to input language.
To ensure the presence of truly incorrect responses (true negatives) among the research questions for the purpose of conducting confusion matrix analysis, one of the answer options in all questions was “none.” Accordingly, the correct answer for at least one-fifth of the questions (10 questions) was “none of the above.” To blind the respondents and reduce potential bias, this information was concealed from both the chatbots and the medical students. Furthermore, the sequence of questions and response options in the study questionnaire was randomized and varied for each participant to ensure the accuracy of the data and to minimize potential bias.
To obtain responses from the chatbots, it was necessary to provide a written instruction to the selected models. To reduce bias and enhance the quality and accuracy of the results, the instruction was prepared collaboratively by two researchers and subsequently reviewed and approved by a third researcher. The instruction, written in Persian, was as follows:
“با توجه به دانش خود در حوزه نورولوژی، به سوال زیر پاسخ دهید”
Which was equivalent to what follows:
“Answer the following question based on your knowledge in neurology.”
Due to the limitation in the number of words that could be entered in the chat interface, it was not possible to present all research questions to the chatbots simultaneously. Therefore, each question was presented individually after refreshing the platform page to clear the temporary memory of the AI and ensure its readiness to respond with maximum accuracy. The responses provided by the chatbots were recorded in a data extraction table, including details such as the name of the chatbot, the date of response, and the answer provided. These records were entered into a Microsoft Word 2020 file by one researcher and subsequently reviewed and verified by another researcher to ensure data accuracy. The data collection from the chatbots was conducted on 8th and 9th of December 2025.
In this phase, the research questionnaire was administered electronically to a sample of medical students. The questionnaires were distributed in hospital classrooms, and students were allocated a suitable amount of time to answer the questions based on their number and complexity. The duration was determined in consultation with two neurology specialists. To facilitate the process and ensure that students applied maximum effort in providing correct responses, the questionnaire was considered as the final exam for their neurology course.
Finally, similar to the previous phase, the collected data were recorded in a Microsoft Office spreadsheet by one researcher. The dataset included variables such as age, gender, academic term, and the responses provided. A second researcher subsequently reviewed and verified the recorded data to ensure its accuracy.
To assess the responses submitted by the chatbots and the students, one of the researchers compared all responses with the gold-standard answers and classified them into two categories: “correct” and “incorrect.” This was done in adherence to the analytical approach utilized within the study (27, 28). At the end of this process, another researcher reviewed the procedure and confirmed its accuracy.
For the analysis and comparison of the accuracy of the chatbots and the students in answering the research questions, a confusion matrix was constructed. Subsequently, four metrics—sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV)—as well as overall accuracy were calculated. The computations were performed by one of the researchers, after which another researcher reviewed and validated the entire calculation process. The calculation methods for these metrics were as follows (28, 29):
True Positive (TP): Correctly selecting an option other than “none.”
False Positive (FP): Incorrectly selecting an option other than “none.”
True Negative (TN): Correctly selecting the “none” option.
False Negative (FN): Incorrectly selecting the “none” option.
The formulas used were:
Sensitivity = TP / (TP + FN).
Specificity = TN / (TN + FP).
Positive Predictive Value (PPV) = TP / (TP + FP).
Negative Predictive Value (NPV) = TN / (TN + FN).
Finally, Accuracy was calculated as:
Accuracy = (TP + TN) / (TP + TN + FP + FN).
Finally, to assess the significance of differences in performance between chatbots and medical students, a paired-samples t-test was conducted. This approach was deemed appropriate, as both groups (chatbots and medical students) responded to the same set of questionnaire items, allowing each student’s score to be paired with the corresponding score from the chatbot. For this purpose, the mean numbers of correct and incorrect responses were calculated separately for chatbots and medical students. In this regard, a coding scheme was applied where 1 denoted correct/true responses and 2 denoted incorrect/false responses. This approach ensured that lower mean scores reflected better performance in questionnaire responses. Additionally, to evaluate the correlation between questionnaire item characteristics and chatbot performance, Pearson correlation coefficients were computed between each item’s responses and its characteristics for every chatbot. All analyses were performed using IBM SPSS Statistics 27.
This study received approval from the Ethics Committee of Birjand University of Medical Sciences (approval code: IR.BUMS.REC.1404.349). All procedures were conducted in accordance with relevant ethical standards and guidelines for research involving human participants. Informed consent was obtained from all participants, their personal information remained confidential throughout and after the study, and participants were free to withdraw at any time.
The research data can be accessed by contacting the corresponding authors of the paper.
Amisha et al. Overview of artificial intelligence in medicine. J. Family Med. Prim. Care. 8 (7), 2328–2331 (2019).
Article CAS PubMed PubMed Central Google Scholar
Lu, Z. et al. Large language models in biomedicine and health: current research landscape and future directions. J. Am. Med. Inform. Assoc. 31 (9), 1801–1811 (2024).
Article CAS PubMed PubMed Central Google Scholar
Telenti, A. et al. Large language models for science and medicine. Eur. J. Clin. Invest. 54 (6), e14183 (2024).
Article PubMed Google Scholar
Huo, B. et al. Large Language Models for Chatbot Health Advice Studies: A Systematic Review. JAMA Netw. Open. 8 (2), e2457879 (2025).
Article PubMed PubMed Central Google Scholar
Du, Q. et al. The Efficacy of Rule-Based Versus Large Language Model-Based Chatbots in Alleviating Symptoms of Depression and Anxiety: Systematic Review and Meta-Analysis J. Med. Internet Res. 27, e78186 (2025). https://doi.org/10.2196/78186
Gunduz, M. E. et al. Evaluating the Readability, Quality, and Reliability of Online Patient Education Materials on Spinal Cord Stimulation. Turk. Neurosurg. 34 (4), 588–599 (2024).
PubMed Google Scholar
Wong, C. et al. Patient use of the internet for health information. Aust Fam Physician. 43 (12), 875–877 (2014).
PubMed Google Scholar
Hanci, V. et al. Youtube as a source of information about Percutan Tracheostomy. Gazi Med. J., 34(4) (2023).
Kara, M. et al. Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. PLOS ONE. 20 (6), e0326351 (2025).
Article CAS PubMed PubMed Central Google Scholar
Daraz, L. et al. Readability of Online Health Information: A Meta-Narrative Systematic Review. Am. J. Med. Qual. 33 (5), 487–492 (2018).
Article PubMed Google Scholar
Özbek, İ. C., Hancı, V. & Özduran, E. Digital Guidance: Quality and Readability Analysis of Artificial Intelligence-Generated Spondyloarthropathy Texts. Turkish J. Osteoporos., 31 (1), 12–18 (2025). https://doi.org/10.4274/tod.galenos.2024.76743
Ozduran, E., Hanci, V. & Erkin, Y. Evaluating the readability, quality and reliability of online patient education materials on chronic low back pain. Natl. Med. J. India. 37 (3), 124–130 (2024).
Article PubMed Google Scholar
Shojaei, P. et al. ChatGPT utilization within the building blocks of the healthcare services: A mixed-methods study. Digit. HEALTH. 10, 20552076241297059 (2024).
Article PubMed PubMed Central Google Scholar
Khosravi, M. et al. Ethical challenges of using artificial intelligence in healthcare delivery: a thematic analysis of a systematic review of reviews. J. Public Health. 33 (11), 2485–2495 (2025).
Article Google Scholar
Khosravi, M. et al. Psychometric properties of an Iranian instrument for assessing adherence to ethical principles in the use of artificial intelligence among healthcare providers. Int. J. Med. Informatics 203, 106019 (2025).
Article Google Scholar
Chen, T. C. et al. Assessment of ChatGPT’s performance on neurology written board examination questions. BMJ Neurol. Open. 5 (2), e000530 (2023).
Article MathSciNet PubMed PubMed Central Google Scholar
Ferrara, P. et al. The role of neurology in the development of community healthcare within the Italian national health service. The position of the Italian society of neurology (SIN). Neurol. Sci. 46 (8), 3363–3375 (2025).
Article PubMed PubMed Central Google Scholar
Janca, A. et al. WHO/WFN Survey of neurological services: a worldwide perspective. J. Neurol. Sci. 247 (1), 29–34 (2006).
Article PubMed Google Scholar
Bagcier, F., Yurdakul, O. V. & Ozduran, E. Top 100 cited articles on ankylosing spondylitis. Reumatismo 72 (4), 218–227 (2021).
Article CAS PubMed Google Scholar
Ros-Arlanzón, P. & Perez-Sempere, A. Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain. JMIR Med. Educ. 10, e56762 (2024).
Article PubMed PubMed Central Google Scholar
Yaman Kula, A. et al. Artificial intelligence versus neurologists: A comparative study on multiple sclerosis expertise. Clin. Neurol. Neurosurg. 250, 108785 (2025).
Article PubMed Google Scholar
Khosravi, M. et al. Artificial Intelligence and Decision-Making in Healthcare: A Thematic Analysis of a Systematic Review of Reviews. Health Serv. Res. Managerial Epidemiol. 11, 23333928241234863 (2024).
Article Google Scholar
Zare, Z. et al. Psychometric evaluation of an instrument measuring artificial intelligence utilization in decision-making domains of healthcare organizations. Sci. Rep. 15 (1), 36698 (2025).
Article ADS CAS PubMed PubMed Central Google Scholar
Shah, S. P. & Heiss, J. D. Artifi cial Intelligence as A Complementary Tool for Clincal Decision-Making in Stroke and Epilepsy. Brain Sci., 14 (3), 228 (2024).
Gorenshtein, A. et al. AI Based Clinical Decision-Making Toolfor Neurologists in the Emergency Department. J. Clin. Med., 14 (17), 6333 (2025).
Pham, T. Ethical and legal considerations in healthcare AI: innovation and policy for safe and fair use. R Soc. Open. Sci. 12 (5), 241873 (2025).
Article ADS PubMed PubMed Central Google Scholar
Shah, R. & Jotterand, F. Large Language Models in Neurology Treatment Decision-Making: a Scoping Review. J. Med. Syst. 49 (1), 115 (2025).
Article PubMed Google Scholar
Inojosa, H. et al. Education Research: Can Large Language Models Match MS Specialist Training? A Comparative Study of AI and Student Responses to Support Neurology Education. Neurol. Educ. 4 (4), e200260 (2025).
Article PubMed PubMed Central Google Scholar
Barrit, S. et al. Specialized large language model outperforms neurologists at complex diagnosis in blinded case-basedevaluation.. Brain Sci., 15 (4), 347 (2025).
Ros-Arlanzón, P. et al. When AI models take the exam: large language models vs medical students on multiple-choice course exams. Med. Educ. Online. 30 (1), 2592430 (2025).
Article PubMed PubMed Central Google Scholar
Sahin, M. C. et al. Beyond human in neurosurgical exams: ChatGPT’s success in the Turkish neurosurgical society proficiency board exams. Comput. Biol. Med. 169, 107807 (2024).
Article PubMed Google Scholar
Schubert, M. C., Wick, W. & Venkataramani, V. Performance of Large Language Models on a Neurology Board-Style Examination. JAMA Netw. Open. 6 (12), e2346721 (2023).
Article PubMed PubMed Central Google Scholar
Guerra, G. A. et al. GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like Questions. World Neurosurg. 179, e160–e165 (2023).
Article PubMed Google Scholar
Bartoli, A. et al. Probing artificial intelligence in neurosurgical training: ChatGPT takes a neurosurgical residents written exam. Brain Spine. 4, 102715 (2024).
Article CAS PubMed Google Scholar
Maiorana, N. V. et al. Large Language Models in Neurological Practice: Real-World Study. J. Med. Internet Res. 27, e73212 (2025).
Article PubMed PubMed Central Google Scholar
Lee, C. et al. Large Language Models Versus Expert Clinicians in Crisis Prediction Among Telemental Health Patients: Comparative Study. JMIR Ment Health. 11, e58129 (2024).
Article PubMed PubMed Central Google Scholar
Khosravi, M. et al. Performance of artificial intelligence large language models (Copilot and Gemini) compared to human experts in healthcare policy making: A mixed-methods cross-sectional study. Health Inf. J.”Bold”>31 (3), 14604582251381269 (2025).
Article Google Scholar
Al-Rahahleh, A. J. et al. Diagnostic performance of four AI tools in pharmacology MCQs: Accuracy, sensitivity, and specificity. PLoS One. 20 (12), e0337688 (2025).
Article CAS PubMed PubMed Central Google Scholar
Most, J. A. et al. Can Multimodal Large Language Models Diagnose Diabetic Retinopathy from Fundus Photos? A Quantitative Evaluation. Ophthalmol. Sci. 6 (1), 100911 (2026).
Article PubMed Google Scholar
Ali, R. et al. Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank. Neurosurgery 93 (5), 1090–1098 (2023).
Article PubMed Google Scholar
Giannos, P. Evaluating the limits of AI in medical specialisation: ChatGPT’s performance on the UK Neurology Specialty Certificate Examination. BMJ Neurol. Open. 5 (1), e000451 (2023).
Article PubMed PubMed Central Google Scholar
Albaqshi, A. et al. Evaluating diagnostic accuracy of large language models in neuroradiology cases using image inputs from JAMA neurology and JAMA clinical challenges. Sci. Rep. 15 (1), 43027 (2025).
Article ADS CAS PubMed PubMed Central Google Scholar
Altunisik, E. et al. Artificial intelligence performance in clinical neurology queries: the ChatGPT model. Neurol. Res. 46 (5), 437–443 (2024).
Article PubMed Google Scholar
Siam, M. K. et al. Benchmarking large language models on the United States medical licensing examination for clinical reasoning and medical licensing scenarios (Scientific Reports, 2025).
Hickman, L., Dunlop, P. D. & Wolf, J. L. The performance of large language models on quantitative and verbal ability tests: Initial evidence and implications for unproctored high-stakes testing. Int. J. Selection Assess. 32 (4), 499–511 (2024).
Article Google Scholar
Levin, C. et al. Can large language models assist with pediatric dosing accuracy? Pediatr. Res. 98 (5), 1760–1765 (2025).
Article PubMed PubMed Central Google Scholar
Roeschl, T. et al. Assessing the Limitations of Large Language Models in Clinical Practice Guideline-Concordant Treatment Decision-Making on Real-World Data: Retrospective Study. JMIRx Med. 6, e74899 (2025).
Article PubMed PubMed Central Google Scholar
Vandenbroucke, J. P. et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Epidemiology 18 (6), 805–835 (2007).
Article PubMed Google Scholar
Download references
ChatGPT-5 was utilized in order to rewrite the text of the manuscript to ensure lack of any grammatical error and enhance readability.
The research was funded by Birjand University of Medical Sciences.
Social Determinants of Health Research Center, Birjand University of Medical Sciences, Birjand, Iran
Mohsen Khosravi
Department of Neurology, Faculty of Medicine, Birjand University of Medical Sciences, Birjand, Iran
Maryam Yousefi-Roobiyat & Zahra Asghari
Razi Hospital, Birjand University of Medical Sciences, Birjand, Iran
Motahareh Nakhaei
School of medicine, Guilan University of Medical Sciences, Rasht, Iran
Fatemeh Khosravi
PubMed Google Scholar
PubMed Google Scholar
PubMed Google Scholar
PubMed Google Scholar
PubMed Google Scholar
MK and MYR conceptualized the study; MK performed the data analysis and wrote the text if the manuscript. MYR and ZA validated the data gathering/analysis, and revised the text of the manuscript. MN and FK conducted the data gathering.
Correspondence to Mohsen Khosravi or Maryam Yousefi-Roobiyat.
The authors declare no competing interests.
was taken from all of the study participants.
All authors of the manuscript have provided their consent for its publication.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Below is the link to the electronic supplementary material.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Reprints and permissions
Khosravi, M., Yousefi-Roobiyat, M., Asghari, Z. et al. Comparison of the performance of ChatGPT-5, Gemini 3, Copilot, Perplexity, and medical students in answering neurology questions: a cross-sectional study. Sci Rep 16, 16070 (2026). https://doi.org/10.1038/s41598-026-47666-5
Download citation
Received:
Accepted:
Published:
Version of record:
DOI: https://doi.org/10.1038/s41598-026-47666-5
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
Advertisement
Scientific Reports (Sci Rep)
ISSN 2045-2322 (online)
© 2026 Springer Nature Limited
Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.