ChatGPT’s Declining Performance Over Time

OpenAI’s chatbot, ChatGPT, powered by artificial intelligence, appears to be experiencing a decline in performance over time, leaving researchers puzzled about the underlying cause. According to a study conducted on July 18 by researchers from Stanford and UC Berkeley, the latest versions of ChatGPT exhibited a significant reduction in their ability to provide accurate answers to the same set of questions compared to just a few months earlier.

Despite their efforts, the study’s authors were unable to pinpoint the exact reasons behind the AI chatbot’s diminishing capabilities. In an attempt to evaluate the reliability of different models of ChatGPT, researchers Lingjiao Chen, Matei Zaharia, and James Zou presented a series of challenges to both ChatGPT-3.5 and ChatGPT-4. These challenges involved solving math problems, answering sensitive questions, writing new lines of code, and performing spatial reasoning tasks based on given prompts.

According to the research, in March ChatGPT-4 was capable of identifying prime numbers with a 97.6% accuracy rate. In the same test conducted in June, GPT-4’s accuracy had plummeted to just 2.4%.

During the period from March to June, both models, ChatGPT-3.5 and ChatGPT-4, experienced a notable decline in their capacity to generate new lines of code. The study revealed that their abilities in this area had substantially deteriorated over time.

Additionally, the researchers noticed a change in ChatGPT’s responses to sensitive questions, with some examples indicating a focus on ethnicity and gender. In the earlier iterations of the chatbot, it would provide extensive reasoning for why it couldn’t answer certain sensitive questions. However, by June, the models had adopted a different approach, offering concise apologies to the user and outright refusing to answer such questions.

“The behavior of the ‘same’ [large language model] service can change substantially in a relatively short amount of time,” the researchers wrote, noting the need for continuous monitoring of AI model quality.

source