Comprehensive Summary
This study examines the effectiveness of publicly available AI platforms in generating Current Procedural Terminology (CPT) codes for orthopaedic foot and ankle procedures. ChatGPT, BingGPT, and Gemini were all chosen as AI search engine platforms to perform comparative analysis on. Each AI was asked what code would be appropriate for a certain common procedure, and the accuracy was then charted in comparison to a practicing foot and ankle surgeon’s response. Cohen Kappa Coefficient analysis revealed the highest agreement AI as BingGPT, followed by Gemini, and then ChatGPT. Though, the ChatGPT version used was GPT-3 as opposed to more up to date models used by Gemini and BingGPT. AI models were found to be moderately effective in identifying the correct CPT code, with an overall accuracy of 44%. The peak accuracy is predicted to be around 50% to 75%, meaning that caution is still advised on AI usage for billing purposes. There is also internal variability in expert opinion on what to code procedures, meaning AI accuracy will always be partially inaccurate.
Outcomes and Implications
This study proposes a novel method of billing patients for standard orthopedic foot and ankle procedures, by generating CPT codes automatically when general procedural information is provided. If AI is able to reach a higher level of accuracy in assigning CPT codes, it may be able to save time for surgeons. This would allow them to focus their efforts on more complex cases and simply review and sign off for standard procedures. The authors do not give a timeline on implementation of AI but believe that there is potential for it to improve the assigning of CPT codes. However, they also highlight the limitations that solely relying on AI is dangerous for practitioners.