Retrieval-augmented generation (RAG) has led to a surge in the number of developers interested in working on retrieval. In this blog post, I share perspectives providing insights and perspectives on the R in RAG.

The case for hybrid search and ranking

Hybrid retrieval and ranking pipelines allow you to combine signals from unsupervised methods (such as BM25) with supervised methods (such as neural rankers). By combining unsupervised and supervised techniques, we have shown that ranking accuracy increases compared to using either method independently. The rise in popularity of hybrid models can be attributed to the lack of the necessary tools, data, time and resources to fine-tune text embedding models specifically for their retrieval tasks. Extensive research and experimentation have shown that hybrid ranking outperforms either method when used alone in a new setting or a new domain with slightly different texts than what the model was trained on.

What is often overlooked in this hybrid search discussion is the ability to perform standard full-text-search (FTS) functionality like exact and phrase matching. Text embedding models are limited by their fixed vocabulary, leading to poor search results for unseen words not in the vocabulary. This is particularly evident in cases such as searching for a product identifier, a phone number, a zip code, or a code snippet, where text embedding models with fixed vocabularies fail. For example, if we look at BERT, one of the most popular language models, its default vocabulary does not include the word 2024.

>>>from transformers import AutoTokenizer
>>>tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

>>>tokenizer.tokenize("2024")
['202', '##4']

>>>tokenizer.encode("2024", add_special_tokens=False)
[16798, 2549]

>>>tokenizer.tokenize("90210")
['90', '##21', '##0']
[3938, 17465, 2692]

We highly recommend this video tutorial for understanding tokenization for language models.

In real-world RAG applications, these search cases are essential. However, the relevancy datasets used to evaluate retrieval and ranking techniques often lack queries of these types. Consequently, when comparing and evaluating retrieval methods on various benchmarks, we only consider limited search use cases.

As more developers address retrieval challenges in the context of RAG, it’s important to remember that text embedding models alone cannot handle simple table stakes search issues.

Multilingual text processing in full-text search

Different languages have unique characteristics that require specific approaches to tokenization, stemming, and normalization. BM25 is suitable for multilingual settings, but it requires attention to diverse languages, character sets and language-specific features.

Tokenization splits text into tokens like words or subwords. It should consider language-specific traits.

Normalization often includes converting text to a consistent case, such as lowercase or uppercase, to eliminate case-sensitive variations. A fun-fact in this respect is that many multilingual text embedding models are built on multilingual tokenizer vocabularies which are case sensitive. That means that the vector representation of “Can” is different from “can”.

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("intfloat/multilingual-e5-large")
>>> tokenizer.tokenize("Can can")
['▁Can', '▁can']
>>> tokenizer.encode("Can can", add_special_tokens=False)
[4171, 831]

These mentioned search text processing techniques have an influence on the shape of the recall and precision curve. In scenarios where high recall is crucial, such as text search, it is generally undesirable for casing to be a decisive factor. However, in other contexts, preserving case may be necessary, especially when distinguishing named entities from other text components.

Vespa as a flexible text search platform integrates linguistic processing components (Apache OpenNLP, Apache Lucene) which provides text processing capabilities for more than 40 languages. Plus, you can roll your own custom linguistic implementation. In addition to the linguistic text processing capabilities, Vespa offers a wide range of matching capabilities like prefix, fuzzy, exact, case sensitive and n-gram catering for a wide range of full-text search use cases.

To chunk or not to chunk

While advancements have introduced LLMs with longer context windows, text embedding models still face limitations in handling long text representations and are outperformed by simple BM25 baselines when used with longer documents. In other words, to produce meaningful text embedding representations for search, we must split longer texts into manageable chunks that can be consumed by the text embedding model.

To address this challenge, developers choosing to work with single-vector databases like Pinecone, have chunked the document into independent retrievable units or rows into the database. This means the original context, the surrounding chunks or other document level metadata is not preserved unless it’s duplicated into the chunk-level retrievable row.

Developers using Vespa, the most versatile vector database for RAG, don’t need to segment the original long document into smaller retrievable units. Multi-vector indexing per document prevents losing the original context and provides easy access to all the chunks from the same document. As a result, developers can retrieve entire documents, not individual chunks.

Another advantage of this representation is that it retains the complete context of the document including metadata. This allows us to employ hybrid retrieval and ranking, which combines signals from both the document level and the chunk level. This technique can be used for candidate retrieval, where relevant documents are identified based on the entire context. The chunk level text embedding representations can then be used to further refine or re-rank the results. Additionally, in the final step of a RAG pipeline, including adjacent chunks or even all the chunks of the document becomes straightforward, provided that the generative model supports a long context window.