Photo by Tamarcus Brown on Unsplash

Where should you begin if you plan to implement search functionality but have not yet collected data from user interactions to train ranking models?

In the first post in the series, we introduced the difference between in-domain and out-of-domain (zero-shot) ranking. We also presented the BEIR benchmark and highlighted cases where in-domain effectiveness does not transfer to another domain in a zero-shot setting.

In this second post in this series, we introduce and evaluate three different Vespa ranking methods on the BEIR benchmark in a zero-shot setting. We establish a new and strong BM25 baseline for the BEIR dataset, which outperforms previously reported BM25 results. We then show how a unique hybrid approach, combining a neural ranking method with BM25, outperforms other evaluated methods on 12 out of 13 datasets on the BEIR benchmark. We also compare the effectiveness of the hybrid ranking method with emerging few-shot methods that generate in-domain synthetic training data via prompting large language models (LLMs).

Establishing a strong baseline

In the BEIR paper, the authors find that BM25 is a strong generalizable baseline text ranking model. Many, if not most, of the dense single vector embedding models trained on MS MARCO labels are outperformed by BM25 in an out-of-domain setting. Quote from BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models:

In-domain performance is not a good indicator for out-of-domain generalization. We observe that BM25 heavily underperforms neural approaches by 7-18 points on in-domain MS MARCO. However, BEIR reveals it to be a strong baseline for generalization and generally outperforming many other, more complex approaches. This stresses the point, that retrieval methods must be evaluated on a broad range of datasets.

What is interesting about reporting BM25 baselines is that there are multiple implementations, variants, and performance tweaks, as demonstrated in Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants. Unfortunately, various papers have reported conflicting results for BM25 on the same BEIR benchmark datasets. The BM25 effectiveness can vary due to different hyperparameters and different linguistic processing methods used in different system implementations, such as removing stop words, stemming, and tokenization. Furthermore, researchers want to contrast their proposed ranking approach with a baseline ranking method. It could be tempting to report a weak BM25 baseline, which makes the proposed ranking method stand out better.

Several serving systems implement BM25 scoring, including Vespa. Vespa’s lexical or sparse retrieval is also accelerated using the weakAnd Vespa query operator. This is important because implementing a BM25 scoring function in a system is trivial, but scoring all documents that contains at least one of the query terms approaches linear complexity. Dynamic pruning algorithms like weakAnd improve the retrieval efficiency significantly compared to naive brute-force implementations that scores all documents matching any of the query terms.

BM25 has two hyperparameters, k1 and b, which impact ranking effectiveness. Additionally, most (14 out of 18) of the BEIR datasets have both title and text document fields, which in a real-production environment would be the first thing that a seasoned search practitioner would tune the relative importance of. In our BM25 baseline, we configure Vespa to independently calculate the BM25 score of both title and text, and we combine the two BM25 scores linearly. The complete Vespa rank profile is given below.

rank-profile bm25 inherits default {
   first-phase {
      expression: bm25(title) + bm25(text)
   }
   rank-properties {
      bm25(title).k1: 0.9
      bm25(title).b: 0.4
      bm25(text).k1: 0.9
      bm25(text).b: 0.4
   }
}

We modify the BM25 k1 and b parameters but use the same parameters for both fields. The values align with Anserini defaults (k1=0.9, b=0.4).

The following table reports nDCG@10 scores on a subset (13) of the BEIR benchmark datasets. We exclude the four datasets that are not publicly available. We also exclude the BEIR CQADupStack dataset because it consists of 12 sub-datasets where the overall nDCG@10 score is found by averaging each sub-dataset’s nDCG@10 score. Adding these sub-datasets would significantly increase the evaluation effort.

The table summarizes the BM25 nDCG@10 results. Vespa BM25 versus BM25 from BEIR paper.

The table above demonstrates that the Vespa implementation has set a new high standard, outperforming reported BM25 baselines on the BEIR benchmark.

Evaluating Vespa ranking models in a zero-shot setting

With the new strong BM25 baseline established in the above section, we will now introduce two neural ranking models and compare their performance with the baseline.

Vespa ColBERT

We have previously described the Vespa ColBERT implementation in this blog post, and we use the same model weights in this work. The Vespa ColBERT model is based on a distilled 6-layer MiniLM model with 22M parameters, using quantized int8 weights (post-training quantization). The model uses only 32 vector dimensions per query and document wordpiece), in contrast to the original ColBERT model, which uses 128 dimensions. Furthermore, we use Vespa’s support for bfloat16 to reduce the per-dimension storage usage from 4 bytes per dimension with float to 2 bytes with bfloat16. We configure the maximum query length to 32 wordpieces, and maximum document length to 180 wordpieces. Both maximum length parameters align with the training and experiments on MS MARCO.

The ColBERT MaxSim scoring is implemented as a re-ranking model using Vespa phased ranking, re-ranking the top 2K hits ranked by BM25. We also compute and store the title term embeddings for datasets with titles, meaning we have two MaxSim scores for datasets with titles. We use a linear combination to combine the title and text MaxSim scores.

The complete Vespa rank profile is given below.

rank-profile colbert inherits bm25 {
   inputs {
      query(qt) tensor<float>(qt{}, x[32])
      query(title_weight): 0.5
   }
   second-phase {
      rerank-count: 2000
	   expression {
	   (1 - query(title_weight))* sum(
	    reduce(
	      sum(
		      query(qt) * cell_cast(attribute(dt), float), x
	      ),
	      max, dt
	    ),
	    qt
	   ) +
	   query(title_weight) * sum(
	    reduce(
	      sum(
		      query(qt) * cell_cast(attribute(title_dt), float), x
	      ),
	      max, dt
	    ),
	    qt
	  )
	}
}

The per wordpiece ColBERT vectors are stored in Vespa using Vespa’s support for storing and computing over tensors.

Note: Users can also trade efficiency versus cost by storing the tensors on disk, or in-memory using paging options. Paging is highly efficient in a re-ranking pipeline, as just a few K tensors values are potentially paged on-demand.

Vespa Hybrid ColBERT + BM25

There are several ways to combine the ColBERT MaxSim with BM25, including reciprocal rank fusion(RRF) which does not consider model scores, just the ordering (ranking) the scores produce. Quote from Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods:

RRF is simpler and more effective than Condorcet Fuse, while sharing the valuable property that it combines ranks without regard to the arbitrary scores returned by particular ranking methods

Another approach is to combine the model scores into a new score to produce a new ranking. We use a linear combination in this work to compute the hybrid score. Like the ColBERT-only model, we use BM25 as the first-phase ranking model and only calculate the hybrid score for the global top-ranking K documents from the BM25 model.

Before combining the scores, we want to normalize both the unbound BM25 and the bound ColBERT score. Normalization is accomplished by simple max-min scaling of the scores. With max-min scaling, scores from any ranking model are scaled from 0 to 1. This makes it easier to combine the two using relative weighting.

Since scoring in a production serving system might be spread across multiple nodes, each node involved in the query will not know the global max scores. We solve this problem by letting Vespa content nodes involved in the query return both scores using Vespa match-features.

A custom searcher is injected in the query dispatching stateless Vespa service. This searcher calculates the max and min for both model scores using match features for hits within the window of global top-k hits ranked by BM25. As with the ColBERT rank profile, we use a re-ranking window of 2000 hits, but we perform feature-score scaling and re-ranking in a stateless custom searcher instead of on the content nodes.

The complete Vespa rank profile is given below. Notice the match-features, which are returned with each hit to the stateless searcher (implementation), which performs the normalization and re-scoring. The first-phase scoring function is inherited from the previously described bm25 rank profile.

rank-profile hybrid-colbert inherits bm25 {
   function bm25() {
	   expression: bm25(title) + bm25(text)
   }

   function colbert_maxsim() {
	   expression {
	      2*sum(
	         reduce(
	            sum(
		            query(qt) * cell_cast(attribute(dt), float) , x
	            ),
	         max, dt
	         ),
	         qt
	      ) +
	      sum(
	         reduce(
	            sum(
		            query(qt) * cell_cast(attribute(title_dt), float), x
	            ),
	         max, dt
	         ),
	         qt
	      )
      }
   }
   match-features {
	   bm25
	   colbert_maxsim
   }
}

​​Results and analysis

As with the BM25 baseline model, we index one of the BEIR datasets at a time on a Vespa instance and evaluate the models. The following table summarizes the results. All numbers are nDCG@10. The best-performing model score per dataset is in bold.

The table summarizes the nDCG@10 results per dataset. Note that MS MARCO is in-domain for ColBERT and Hybrid. Average nDCG@10 is only computed for zero-shot and out-of-domain datasets.

As shown in the table above, in a in-domain setting on MS MARCO, the Vespa ColBERT model outperforms the BM25 baseline significantly. The resulting nDCG@10 score aligns with reported MRR@10 results from previous work using ColBERT in-domain on MS MARCO. However, mixing the baseline BM25 using the hybrid model on MS MARCO evaluation hurts the nDCG@10 score, as we combine two models where the unsupervised BM25 model is significantly weaker than the ColBERT model.

The Vespa ColBERT model underperforms BM25 on out-of-domain datasets, especially CLIMATE-FEVER. The CLIMATE-FEVER dataset has very long queries (avg 20.2 words). The long questions challenge the ColBERT model, configured with a max query length of 32 wordpieces in the experimental setup. Additionally, the Vespa ColBERT model underperforms reported results for the full-sized ColBERT V2 model using 110M parameters and 128 dimensions. This result could indicate that the compressed (in the number of dimensions) and model distillation have a more significant negative impact when applied in a zero-shot setting compared to in-domain.

These exceptions aside, the data shows that the unique hybrid Vespa ColBERT and BM25 combination is highly effective, performing the best on 12 of 13 datasets. Its average nDCG@10 score improves from 0.453 to 0.481 compared to the strong Vespa BM25 baseline.

To reproduce the results of this benchmark, follow the open-sourced instructions.

Comparing hybrid zero-shot with few-shot methods

To compare the hybrid Vespa ranking performance with other models, we include the results reported in Promptagator: Few-shot Dense Retrieval From 8 Examples from Google Research.

Generating synthetic training data in-domain via prompting LLMs is a recent emerging Information Retrieval(IR) trend also described in InPars: Data Augmentation for Information Retrieval using Large Language Models.

The basic idea is to “prompt” a large language model (LLM) to generate synthetic queries for use in training of in-domain ranking models. A typical prompt include a few examples of queries and relevant documents, then the LLM is “asked” to generate syntetic queries for many of the documents in the corpus. The generated syntetic query, document pairs can be used to train neural ranking models. We include a quote describing the approach from the Promptagator paper:

Running the prompt on all documents from DT, we can create a large set of synthetic (q, d) examples, amplifying the information from few examples into a large synthetic dataset whose query distribution is similar to true task distribution QT and query-document pairs convey the true search intent IT. We use FLAN (Wei et al., 2022a) as the LLM for query generation in this work. FLAN is trained on a collection of tasks described via instructions and was shown to have good zero/few-shot performance on unseen tasks. We use the 137B FLAN checkpoint provided by the authors.

The Promptagator authors report results on a different subset of the BEIR datasets (excluding Quora and Natural Questions). In the following table we compare their reported results on the same BEIR datasets used in this work. We also include the most effective single-vector representation model (TAS-B) from the BEIR benchmark (zero-shot).

Vespa ranking model comparison with few-shot models and singe-vector TAS-B (zero-shot). The PROMPTAGATOR results are from table 2 in the paper.

The dense TAS-B model underperforms both the BM25 baseline and the hybrid model. This result is in line with other dense models trained on MS MARCO; dense single-vector representation models struggle with generalization in new domains.

The PROMPTAGATOR single-vector representation model (110M parameter) performs better than the zero-shot Vespa hybrid model. Still, given that it has performed in-domain adoption, we don’t think the difference is that significant (0.456 versus 0.478). Furthermore, we could also adapt the hybrid model on a per-dataset basis, for example, by adjusting the relative importance of the title and text fields. Interestingly, PROMPTGATOR reports a BM25 baseline nDCG@10 score of 0.418 across these datasets, which is considerably weaker than the strong Vespa BM25 baseline of 0.436.

We also include the PROMPTGATOR re-ranking model, a cross-encoder model with another 110M parameters, to re-rank the top-200 results from the retriever model. This model outperforms all other methods described in this blog post series.

There is also exciting work (InPars v2) using LLMs to generate synthetic training data that report strong cross-encoder model results on BEIR, but with models of up to 3B parameters, which makes it impractical and costly in production use cases.

Cross-encoder models can only be deployed as a re-ranking phase as they input both the query and the document and is more computationally intensive than other methods presented in this blog post. Nevertheless, the computationally inexpensive Vespa hybrid model could be used as a first-phase retriever for cross-encoder models. We described cross-encoder models in Vespa in part four in our Pretrained Transformer Language Models for Search blog post series.

Deploying hybrid ranking models to production

We’ve made everything you need to deploy this solution available.

This research, evaluating Vespa zero-shot models on BEIR began with COVID-19 Open Research Dataset (CORD-19). We have indexed the complete, final version of the CORD-19 dataset on https://cord19.vespa.ai/. You can select between all three ranking models described in this blog post. The demo search also includes result facets, result pagination, result snippets, and highlighting of matched query terms. Essentially, everything you expect from a search engine implementation.

The Vespa app is open-source and deployed on Vespa Cloud, and the app can also be run locally using the open-source Vespa container image. The Vespa ColBERT model is CPU-friendly and does not require expensive GPU/TPU acceleration to meet user-serving latency requirements. The end-to-end retrieval and ranking pipeline, including query encoding, retrieval, and re-ranking, takes less than 60 ms.

Summary

In this blog post in a series on zero-shot ranking, we established a strong BM25 baseline on multiple BEIR datasets, improving over previously reported results. We believe that without a strong BM25 baseline model, we can overestimate the neural ranking progress, especially in a zero-shot setting where neural single vector representations struggle with generalization.

We then introduced a unique hybrid ranking model, combining ColBERT with BM25 and setting a new high bar for efficient and effective zero-shot ranking. We also compared this unique model’s effectiveness with much larger models that use few-shot in-domain adoption techniques involving billion-sized LLMs.

Importantly, all the results presented in this blog post are easily reproduced using the open-sourced Vespa app, which is deployed to production and available at https://cord19.vespa.ai/.

For those interested in learning more about hybrid search in a zero-shot setting, we highly recommend two Vespa related talks presented at Berlin Buzzwords 2022.