Do Doctors Need a DOS of Their Own Medicine?

Code Blue Screen

A medical study was published last week that found doctors co-diagnosing using AI were more likely to get the diagnosis wrong because they disregarded the machine when it questioned their expertise. 

This resistance to computer input has itself been given the diagnosis of “algorithmic aversion” – a cognitive risk factor most likely to be associated with people who are highly experienced or expert in their field.

The report, Large Language Model Influence on Diagnostic Reasoning (Goh et al., 2024) observed that when diagnosing medical conditions, Chat GPT-4 alone significantly outperformed doctors who were using GPT-4. 

This output goes against the prevailing body of research which states that human + AI is the most effective collaboration, rather than AI alone or human alone.

For example, a 2023 Harvard report established workers using ChatGPT outperformed those who did not by 40% (Dell’Acqua et al., 2023). Similarly, a paper from MIT uncovered a 37% increase in productivity for workers using AI (Noy and Zhang, 2023) and GitHub CEO Thomas Dohmke says that coders using AI in their work are 55% faster (Scheffler, 2023).

Calling Dr Beep

Dr. Adam Rodman, one of the authors of the medical report explained in a podcast interview (Hard Fork, 00:31:00) that the doctors did not want to accept that the AI could be better than them at diagnosis, and would second guess what the AI suggested, and end up getting the diagnosis wrong. 

Dr. Rodman explained that there were two factors at work in the outcome of their study, one was a lack of experience working with the AI technology and not being experienced at using detailed prompts to get good responses. But, the other reason was resistance by some doctors to having their medical opinion questioned by an AI chatbot.

“Some people didn’t quite know how to use a Language Model to get the most use out of it, so training is one factor” he said. “Number two though, when you look at the data, people liked it when the AI model said “oh, this is your idea, these are the things I agree with”. But when the AI model said “Hey Man! You might be wrong, these things don’t fit” they disregarded that”.

A hard pill to swallow



This type of resistance to AI input has been labelled ‘algorithmic aversion’ by Ethan Mollick, Professor of Management at Wharton, who specialises in researching how AI impacts entrepreneurship and innovation.

Mollick also suggests that the majority of us don’t really understand what AI does and we mostly use it like a search engine and get frustrated by the results.

He said if you’re not used to working with a chat bot “AI systems are surprisingly hard to get a handle on, resulting in a failure to benefit from their advice” (Mollick, 2024). 

"The future of medicine lies not in choosing between human expertise and artificial intelligence, but in learning to leverage both".

An AI a day keeps the errors away!

Artificially intelligent chatbots are a very different tool from the search engines we're used to. Chatbots are more like an incredibly forgetful but endlessly knowledgeable intern. You have to explain very carefully what you want, and understand that the chatbot doesn't necessarily remember any previous conversations and has absolutely no context. 

"Treat AI like an infinitely patient new coworker" suggests Ethan Mollick, "a coworker who forgets everything you tell them each new conversation, one that comes highly recommended, but whose actual abilities are not that clear" (Mollick, 2024).

In order to learn how to work efficiently with these amnesiac colleagues Mollick advises spending around 10 hours simply talking with an AI chatbot to get used to how it works.

So maybe what the doctors need is not so much an apple a day, but a regular dose of  AI chatbot exposure, to help them to get over their aversion to this potentially life-saving tech.