When you are working on the frontier of applied generative AI, it’s useful to get occasional reality checks from complete outsiders to the field. Do our intuitions and internal evaluations hold up when meeting the real world?
That’s why we found this study so interesting: It’s a comparison of the outcomes of using various AI Search engines carried out by journalists and published in Columbia Journalism Review. Their key result is captured in this figure:
Each square is a task, green ones were completed correctly, and red ones incorrectly. As you can see, Perplexity is the clear winner, and only ChatGPT Search is even playing in the same league! This matches what we thought we knew, but it is nice to have it validated by outsiders evaluating solutions in the way that makes most sense to them.
How can Perplexity crush it when they are using the models of their competitors?
While Perplexity does offer some model fine-tunes, they mostly offer the models developed by their competitors, so how can they beat them so handily, where lies the big difference?
Since you are here, you likely know the answer: Search relevance.
- For most tasks, no amount of model intelligence helps if the model isn't being given the information it needs to solve the problem it is facing.
Perplexity has made the necessary investment into relevance and scale to be able to retrieve exactly the information the model needs to solve most problems, and they have done it on the only platform that lets you achieve great quality at scale: Vespa.ai.
The Vespa platform gives you flexibility to build any kind of solution where you need to find some data and make inferences in it really fast, and then scale it to any size. However, lots of freedom can be daunting, and even the biggest teams we support initially find it challenging to make choices when building the kind of state-of-the-art RAG applications Vespa enables.
That’s why we decided to collect everything we regard as best practices for RAG into a single open-source deployable sample application: The RAG Blueprint. By cloning and modifying this application, you can be assured that you’ll have an application that delivers world-class quality and performance at low cost, and which can seamlessly scale to any amount of data and traffic you’ll realistically need to handle.
You can get started with the RAG Blueprint by following the RAG Blueprint tutorial.
