Optimizing Ophthalmology Education via ChatBot-Generated Materials: Readability Analysis of AI-generated Patient Education Materials and North American Neuro-Ophthalmology Society (NANOS) Patient Brochures

Background: It is crucial that patients are offered concise patient education materials (PEMs) to facilitate an enhanced understanding of diagnoses and treatment options. As the average American reads at an 8th grade reading level, the AMA suggests that PEMs be written at a 6th grade level. Despite significant efforts made by NANOS to provide easily accessible webpages for different diagnoses, treatments, and surgeries, the readability of such materials exceeds these recommended guidelines. AI chatbots such as Chat-GPT and Google’s Bard can generate sophisticated responses to free-text input and modifying responses to user feedback. When prompted appropriately, such technology offers promising application to fill the need for more comprehensible ophthalmological PEMs. As such, the aim of this study is to compare the readability of North American Neuro-Ophthalmology Society (NANOS) patient brochures to that of PEMs generated by the AI-chat bots ChatGPT 3.5 and Google Bard.

Methods: PEMs on 13 common neuro-ophthalmology topics were generated by ChatGPT 3.5 and Google Bard, with and without a 6th grade reading level prompt. The PEMs were analyzed using seven readability metrics: Flesch Reading Ease score, Gunning Fog Index, Flesch-Kincaid Grade Level, Coleman–Liau Index, SMOG Index Score, Automated Readability Index, and Linsear Write Readability Score. Each AI-generated PEM was compared with the equivalent NANOS PEM.

Results: When comparing ChatGPT generated PEMs to NANOS brochures, unprompted ChatGPT PEMs consistently had lower readability scores (Flesch-Reading Ease Score NANOS 49.5 ± 3.3; ChatGPT 43.8 ± 8.8, p = 0.04) and higher grade level-equivalent readability scores (SMOG index NANOS 9.7 ± 0.5; ChatGPT 10.4 ± 1.0, p = 0.04), indicating that this material may be the most difficult to comprehend in its unprompted form. On the other hand, Bard generated PEMs that corresponded to a lower grade level compared to NANOS brochures (Linsear-Write Formula NANOS 12.1 ± 1.5; Bard 10.6 ± 2.2), though this difference was non-significant. However, when prompted to produce PEMs at a 6th grade reading level, both ChatGPT and Bard possessed far superior readability scores across all readability metrics compared to NANOS texts. The grade-level equivalent averaged around 8 for both ChatGPT and Google Bard, whereas NANOS brochures corresponded to an 11th grade level.

Conclusion: This study offers compelling evidence that, when prompted appropriately, AI chat bots can generate comprehensible PEMs for patients on a variety of neuro-ophthalmology topics. Use of this technology in a clinical setting may offer patients with succinct and readable information on complex topics and can guide patient-oriented decision making. Furthermore, use of AI chat bots in this manner may be clinically beneficial in satellite centers where there is a scarcity of neuro-ophthalmologists by assisting triage of patients for neuro-ophthalmology referrals.