Loading…
Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery
Purpose ChatGPT is a large language model trained on a large dataset covering a broad range of topics, including the medical literature. We aim to examine its accuracy and reproducibility in answering patient questions regarding bariatric surgery. Materials and methods Questions were gathered from n...
Saved in:
Published in: | Obesity surgery 2023-06, Vol.33 (6), p.1790-1796 |
---|---|
Main Authors: | , , , , , , , , , , , |
Format: | Article |
Language: | English |
Subjects: | |
Citations: | Items that this one cites Items that cite this one |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Summary: | Purpose
ChatGPT is a large language model trained on a large dataset covering a broad range of topics, including the medical literature. We aim to examine its accuracy and reproducibility in answering patient questions regarding bariatric surgery.
Materials and methods
Questions were gathered from nationally regarded professional societies and health institutions as well as Facebook support groups. Board-certified bariatric surgeons graded the accuracy and reproducibility of responses. The grading scale included the following: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. Reproducibility was determined by asking the model each question twice and examining difference in grading category between the two responses.
Results
In total, 151 questions related to bariatric surgery were included. The model provided “comprehensive” responses to 131/151 (86.8%) of questions. When examined by category, the model provided “comprehensive” responses to 93.8% of questions related to “efficacy, eligibility and procedure options”; 93.3% related to “preoperative preparation”; 85.3% related to “recovery, risks, and complications”; 88.2% related to “lifestyle changes”; and 66.7% related to “other”. The model provided reproducible answers to 137 (90.7%) of questions.
Conclusion
The large language model ChatGPT often provided accurate and reproducible responses to common questions related to bariatric surgery. ChatGPT may serve as a helpful adjunct information resource for patients regarding bariatric surgery in addition to standard of care provided by licensed healthcare professionals. We encourage future studies to examine how to leverage this disruptive technology to improve patient outcomes and quality of life.
Graphical Abstract |
---|---|
ISSN: | 0960-8923 1708-0428 |
DOI: | 10.1007/s11695-023-06603-5 |