We Evaluated ChatGPT vs. Google on 500 Search Queries
ChatGPT crushes Google on coding queries and ties it on general informational queries – despite not being optimized for a search experience at all. OpenAI’s existential threat to Google, and reach out to team@surgehq.ai if you’re interested in exploring the data.
ChatGPT vs. Google
- Imagine you want to recursively delete all the Python files in a folder.
- Google misinterprets your query and returns a 404 error, prompting you to switch to chatGPT
- The sexy upstart is hyper-customized to your.py need, and provides the perfect tip you never thought to ask.
ChatGPT Pros
- Synthesizes information from a variety of sources into a single, coherent whole
- Minimal interface
- Ability to understand complex queries
Cons
- Hallucinations and inaccuracies
- The AI struggled with anything that wasn’t very common knowledge
Google’s Existential Threat
- Google’s 24 years of search expertise have been toppled by a new technology that soon even small startups will be able to use
- Newer search engines like Neeva, You.com, Kagi, and Bing already move faster, with the freedom to explore new products and UIs that Google can’t
- Imagine AI models just as smart as Google’s or smarter! – in their hungrier, more product-focused hands
Large Language Models and the Google-Killer
- ChatGPT is the definition of the future of Search.
- Twitter has been running amok with examples of how it crushes Google
- But is this representative?
- Large language models might be the Google killer.
Human evaluation
- 100 users were asked to pull up their search history and extract their 5 most recent “informational” queries
- They reissued the same query on Google and posed it to ChatGPT
- The results? Despite not being optimized for a search experience at all, chatGPT already matches or slightly beats Google’s performance
- Google loses out 70% of the time on coding-specific queries
- The Chatbot was better in terms of delivering a concise answer that anyone (myself included) could unambiguously digest.
- Google, full of empty filler and irrelevant ads, was not even close to as nice as the Chatbot.
