ChatGPT Product Recommendations Shift Significantly with Search Enabled, Study Finds

A study by Jeff Oxford, founder and CEO of Visibility Labs, has found that 80.2% of product recommendations generated by ChatGPT change when the AI's search feature is activated. The research involved 20,000 responses, derived from testing 1,000 product-recommendation prompts ten times each with search enabled and ten times with search disabled.

The study identified that only 19.8% of products recommended when search was disabled also appeared in recommendations generated with search enabled. This shift was notable even for products consistently suggested by ChatGPT without web access; among those that appeared in 100% of search-disabled responses, only 15.8% were also present when search was turned on.

The research also explored the relationship between products mentioned in ChatGPT's cited sources and their recommendation frequency. A 0.4 Pearson correlation was observed between cited-source mentions and a product's "Visibility Score," defined as the percentage of runs in which a product appeared for a given prompt. Products mentioned more frequently in cited sources tended to achieve higher Visibility Scores. However, the study noted that this analysis did not establish a causal link.

Furthermore, the presence of search functionality appeared to narrow the scope of recommendations. ChatGPT responses with search enabled contained an average of 5.2 products, compared to 6.2 products when search was disabled. Across ten runs of each prompt, an average of 19 unique products per prompt were returned with search enabled, versus 21.8 unique products with search disabled.

(Source: Search Engine Land)