This was a long requested feature on our tracker (see GitHub issue #28761).

The 2 GB limitation

ONNX uses Google’s Protocol Buffers as its serialization format. Protobuf has a hard limit of 2 GB on message size. For smaller models, this is not a problem — all tensor data (the model weights) is embedded directly in the .onnx file, making it self-contained.

As models grow larger, they inevitably hit this limitation. For a model exceeding 2 GB, ONNX tooling splits it into two parts:

• A small .onnx file containing the model graph structure (typically a few hundred KB to a few MB).
• One or more external data files (commonly named .onnx_data) containing the actual tensor weights.

Note that reduced-precision variants of these models (INT8, FP16, etc.) are often small enough to fit in a single self-contained .onnx file. The external data split primarily affects the full-precision versions.

Previously, if you pointed a Vespa embedder at a model with external data files, ONNX Runtime would fail to load it because the data files were not available alongside the model file.

What changed

Vespa embedders now automatically handle ONNX models with external data files. When you configure an embedder with a URL pointing to an .onnx file, Vespa inspects the model to check whether it references any external data files. If it does, Vespa downloads those files automatically before loading the model.

This feature is available starting from Vespa 8.544.

How to use it

Here is an example using EmbeddingGemma 300M, which uses external data:

<container id="default" version="1.0">
  <component id="gemma" type="hugging-face-embedder">
    <transformer-model
      url="https://huggingface.co/onnx-community/embeddinggemma-300m-ONNX/resolve/main/onnx/model.onnx"/>
    <tokenizer-model
      url="https://huggingface.co/onnx-community/embeddinggemma-300m-ONNX/resolve/main/tokenizer.json"/>
    <max-tokens>2048</max-tokens>
    <prepend>
      <query>task: search result | query: </query>
      <document>title: none | text: </document>
    </prepend>
  </component>
</container>

If you are deploying to Vespa Cloud, you can also use models from the Vespa Model Hub that use external data. For example, the Multilingual-E5-large model (will be available on Vespa Cloud 8.668+):

<container id="default" version="1.0">
  <component id="e5" type="hugging-face-embedder">
    <transformer-model model-id="multilingual-e5-large"/>
    <max-tokens>512</max-tokens>
    <prepend>
      <query>query: </query>
      <document>passage: </document>
    </prepend>
  </component>
</container>

This works with our ONNX-based embedders:

• hugging-face-embedder
• colbert-embedder
• splade-embedder

It’s also possible to use private models — authentication tokens are propagated when downloading external data files.

Current limitations

There are a few constraints to be aware of:

• Embedders only. Models used directly in ranking expressions must still be self-contained and under 2 GB.

• URL-referenced or Model Hub models only. Models bundled in the application package using the path attribute do not support external data. Models referenced via url or model-id (Vespa Cloud) are supported.

• External data files must be co-located with the model. The external data files are resolved relative to the model URL. They must be in the same directory (or a subdirectory) as the .onnx file.

See the ONNX model documentation for the full list of requirements.

If you need more extensive support for ONNX models with external data — for example in ranking expressions — feel free to file an issue.

What if you could drop that to $0?

That’s the promise of asymmetric retrieval: embed your documents with the best model money can buy, then embed queries with a tiny model running locally — for free. Voyage AI’s new voyage-4 family is the first to make this practical, and Vespa now has native support for it.

The asymmetric insight

The conventional approach is to use the same embedding model for documents and queries. Same model, same vector space. But it ignores a fundamental asymmetry.

Document embedding is a one-time cost. You embed each document once at indexing time, and it’s not latency-sensitive — whether it takes 10ms or 500ms doesn’t matter because no user is waiting. You can throw the biggest, most accurate model at it and take your time.

Query embedding is the opposite. It’s on the critical path of every single request, continuously, at scale. It needs to be fast, and at 10K QPS the cost dwarfs everything else.

Why use the same model for both?

Asymmetric retrieval splits these two concerns:

1. Documents — Embed once with voyage-4-large. Best accuracy, API-based, no rush.
2. Queries — Embed continuously with voyage-4-nano. Tiny, local, free.

This works because all four models in the Voyage 4 family — voyage-4-large, voyage-4, voyage-4-lite, and voyage-4-nano — produce compatible embeddings in a shared vector space.

It also means you can upgrade your query model independently. Start with voyage-4-nano for cost, move to voyage-4-lite for quality — without re-embedding a single document.

The shared embedding space opens up document-side flexibility too. In a multi-tenant system, you could use different models for different tiers — voyage-4-large for premium customers who need the best retrieval quality, voyage-4-lite for cost-sensitive tenants — all searchable with the same query model. Same index, same query path, different quality/cost tradeoffs per tenant.

The numbers

Cost

Let’s be concrete about the 10K QPS scenario:

• 10,000 QPS × 30 tokens = 300,000 tokens/sec
• 300,000 × 60 × 60 × 24 × 30 = ~777 billion tokens/month
• At $0.02/1M tokens ≈ $15,500/month for query embeddings via API

With voyage-4-nano running locally on the Vespa container: $0/month. The model runs as part of the serving infrastructure you’re already paying for.

Latency

API calls add network round-trips. Local inference on voyage-4-nano runs in single-digit milliseconds on CPU.

Quality

Voyage 4 is state-of-the-art. On the RTEB benchmark (29 retrieval datasets, NDCG@10), voyage-4-large beats the competition:

And asymmetric retrieval — querying with a smaller model against voyage-4-large document embeddings — preserves retrieval quality across medical, code, web, finance, and legal domains.

Storage

Binary quantization gives you a 16x memory reduction over bfloat16 — 2048-dim vectors go from 4,096 bytes to 256 bytes. The full-precision floats are still used for second-phase reranking, paged from disk only when needed. For a deeper dive on quantization tradeoffs, see Embedding Tradeoffs, Quantified.

Why this matters at scale

Cost and quality are table stakes. The real question for large-scale systems is: does this work in production?

Independent scaling

Vespa separates stateless containers (where embedding runs) from content clusters (where data lives). This means you can scale query embedding capacity independently from storage. Need more QPS? Add container nodes. More documents? Add content nodes. They don’t interfere.

No external API on the query path

This is the underrated benefit. With asymmetric retrieval, the query embedding model runs locally inside Vespa — your critical search path has zero dependency on an external API.

That matters when:

• The API goes down. Every embedding API has outages. If your query path depends on one, your search goes down with it.
• You get rate-limited. Traffic spikes don’t care about your API quota. A sudden 3x in query volume means dropped requests — or queued requests that blow your latency budget.
• You need to scale fast. Adding Vespa container nodes takes minutes. Negotiating higher API rate limit may take days. On Vespa Cloud, autoscaling handles traffic spikes automatically — container clusters are stateless and scale up quickly.

Keeping the query path self-contained turns your search system from “works when everything is up” into “works, period.”

Two-phase ranking

Binary vectors are fast — Vespa can do ~1 billion hamming distance calculations per second. But binary quantization loses precision. Vespa’s phased ranking recovers it:

1. First phase: Hamming distance on binary embeddings. Fast, cheap, scans the full index.
2. Second phase: Float dot-product on the top 2,000 candidates. Accurate, but only touches a bounded set of vectors paged from disk.

This gives you the speed of binary search with the accuracy of full-precision reranking.

Enterprise-proven

This isn’t theoretical. Vespa runs search and recommendation at Spotify, Yahoo, and Perplexity — billions of documents, thousands of QPS, sub-100ms latency. The architecture handles it.

How to set this up

Here’s the complete Vespa configuration for asymmetric retrieval with Voyage AI.

Schema

Two embedding fields — binary for fast retrieval, float for accurate reranking:

schema doc {
  document doc {
    field id type string {
      indexing: summary | attribute
    }
    field text type string {
      indexing: index | summary
    }
  }

  field embedding_float type tensor<bfloat16>(x[2048]) {
    indexing: input text | embed voyage-4-large | attribute
    attribute {
      distance-metric: prenormalized-angular
      paged
    }
  }

  field embedding_binary type tensor<int8>(x[256]) {
    indexing: input text | embed voyage-4-large | attribute
    attribute {
      distance-metric: hamming
    }
  }
}

The paged attribute on embedding_float tells Vespa to keep these vectors on disk, paging them into memory only during second-phase reranking. The binary embeddings stay in memory for fast first-phase retrieval.

Embedders (services.xml)

Two embedders — one API-based for documents, one local for queries:

<container id="default" version="1.0">
  <component id="voyage-4-large" type="voyage-ai-embedder">
    <model>voyage-4-large</model>
    <api-key-secret-ref>apiKey</api-key-secret-ref>
    <dimensions>2048</dimensions>
    <batching max-size="20" max-delay="20ms"/>
  </component>

  <component id="voyage-4-nano" type="hugging-face-embedder">
    <transformer-model model-id="voyage-4-nano-int8"/>
    <tokenizer-model model-id="voyage-4-nano-vocab"/>
    <max-tokens>32768</max-tokens>
    <pooling-strategy>mean</pooling-strategy>
    <normalize>true</normalize>
    <prepend>
      <query>Represent the query for retrieving supporting documents: </query>
    </prepend>
  </component>
</container>

The voyage-ai-embedder handles vector quantization automatically — it infers the target precision from the destination tensor type. bfloat16 fields get full-precision embeddings; int8 fields get binary representations.

The hugging-face-embedder runs voyage-4-nano locally. No API calls, no rate limits, no cost. Both model references (voyage-4-nano-int8, voyage-4-nano-vocab) resolve via the Vespa Model Hub.

A note on “quantization” — two different things. The voyage-4-nano-int8 in the model-id refers to model weight quantization: the ONNX model file uses INT8 weights instead of FP32, which makes inference 2-3x faster on CPU with negligible quality loss. This is about how the model itself is stored and executed. The embedder still produces full-precision float vectors as output. Vector quantization is a separate concern — it’s about the precision of the output embeddings you store and search over (bfloat16, int8/binary, etc.). That’s controlled by the tensor type in your schema field, not the model format. These are independent knobs: you can run an INT8-quantized model that outputs float vectors, then store them as binary. For a deeper dive with benchmarks on both, see Embedding Tradeoffs, Quantified.

Rank profile

Two-phase ranking: hamming distance first, float reranking second:

rank-profile binary-with-rerank {
  inputs {
    query(q_float) tensor<float>(x[2048])
    query(q_bin) tensor<int8>(x[256])
  }

  function binary_closeness() {
    expression: 1 - (distance(field, embedding_binary) / 2048)
  }

  function float_closeness() {
    expression: reduce(query(q_float) * attribute(embedding_float), sum, x)
  }

  first-phase {
    expression: binary_closeness
  }

  second-phase {
    expression: float_closeness
    rerank-count: 2000
  }
}

Querying

Both query tensors are produced by the local voyage-4-nano embedder:

yql=select * from doc where {targetHits: 100}nearestNeighbor(embedding_binary, q_bin)
&ranking=binary-with-rerank
&input.query(q_bin)=embed(voyage-4-nano, "your query here")
&input.query(q_float)=embed(voyage-4-nano, "your query here")
&hits=10

The nearest neighbor search runs on the binary field for speed, while the rank profile handles two-phase scoring.

For a complete runnable example with pyvespa, see the Voyage AI embeddings notebook.

Wrapping up

Asymmetric retrieval makes the most sense when:

• High QPS — The cost savings scale linearly. At 10K QPS, you’re saving $15.5K/month. At 100K QPS, it’s $155K.
• Large corpus — Documents are embedded once, so the large model cost is amortized. The bigger the corpus, the more you benefit from cheap queries.
• Latency-sensitive — Local inference eliminates network round-trips.

When a single model is the better choice:

• Low volume and latency-tolerant — At 10 QPS, the API cost is ~$15/month and the network round-trip doesn’t matter. One model is simpler to operate.
• Quality above all else — Using voyage-4-large for both documents and queries gives you the best possible retrieval quality. If you can afford the API cost and latency, symmetric with the top model is hard to beat.

The Voyage 4 family and Vespa’s native integration make asymmetric retrieval practical for the first time. Embed documents with the best model available, query with a tiny local model, and let phased ranking close the quality gap.

Resources:

• Voyage AI embeddings notebook — Full runnable example
• Embedding documentation — Configuring embedders in Vespa
• Binary quantization guide — Deep dive on binarization
• Phased ranking — Multi-phase ranking architecture
• Voyage 4 announcement — Model family details and benchmarks

For those interested in learning more about Vespa, join the Vespa community on Slack to exchange ideas, seek assistance from the community, or stay in the loop on the latest Vespa developments.

“95% of our retrieval is done by AI agents.” - Sergio Prada, Co-Founder & CTO, Metal

Metal needed a retrieval foundation that could evolve as fast as their product, without hitting a wall.

Introduction

Private equity firms manage vast amounts of unstructured data, including deal documents, expert call transcripts, financial statements, CRM records, and more. The challenge isn’t simply accessing this information. It’s connecting and understanding it, in context, across the investment lifecycle.

Metal AI was built to address this challenge. Its purpose-built institutional intelligence platform, used by established private equity firms transforms fragmented historical and live deal data into a living system of record that drives conviction at every stage of the investment lifecycle.

To deliver this vision at scale, Metal leverages Vespa.ai as its core retrieval layer, powering entity relationships, advanced ranking, and real-time context-aware retrieval across complex investment data.

The Need for Relationship-Driven Retrieval

As Metal’s product evolved, the limitations of traditional retrieval systems became clear.

Early architecture supported basic document search, but private equity workflows aren’t document-centric. They are entity- and relationship-driven. The enduring edge in private equity lies in drawing on decades of deal history, portfolio outcomes, and institutional knowledge. When that depth of experience surfaces reasoning and connections across time, every investment decision carries greater conviction.

Most traditional vector stores and search engines are fundamentally document-first. They index text, return similar passages, and rely primarily on semantic similarity or keyword matching. But for Metal’s use case, relevance requires more:

• Understanding which answer is the most recent and legally approved

• Identifying which company a metric belongs to

• Connecting meetings to prior diligence activity

• Applying business logic alongside semantic similarity

As Metal introduced more advanced workflows, like DDQ automation and agent-driven retrieval, the gap widened. Traditional systems struggle to:

• Combine semantic similarity with recency and compliance rules within ranking

• Support evolving data models without significant rework

• Query across multiple object types in a unified way

• Serve as a foundation for structured, iterative queries issued by AI agents

Layering custom logic on top of limited retrieval infrastructure would have created increasing technical debt, and each new entity type or ranking rule risked architectural compromise.

Metal needed a retrieval foundation that could evolve with the product, not constrain it.

Choosing a Retrieval Layer without Limits

Metal wasn’t simply selecting a search engine. They were selecting a long-term retrieval architecture.

Several capabilities distinguished Vespa:

• Multi-entity modeling: Vespa supports multiple object types, like documents, people, activities, and financial data, as well as the relationships between them. This aligned with how Metal structures institutional knowledge.

• Advanced ranking and filtering: Vespa can combine semantic similarity with structured filters like recency and business rules, enabling Metal to tailor retrieval to specific workflows.

• Flexibility without re-architecture: New object types can be introduced without migrating existing data or rebuilding the system.

• Operational simplicity: Moving to Vespa Cloud enabled the team to focus engineering capacity on product innovation instead of infrastructure.

These capabilities give Metal the ability to shape retrieval around business logic, rather than forcing business logic to adapt to infrastructure limitations.

“Our competitors focus on documents. With Vespa, we can focus on the full picture: companies, people, activities, and how they relate.” - Sergio Prada, Co-Founder & CTO, Metal

Architecture in Action

Metal treats retrieval as part of an AI agent orchestration layer, not just a standard search box.

When a user or agent asks a question like, “What’s this company’s EBITDA?”, the query is first interpreted by an AI agent. Rather than issuing a single plain-text search, the agent:

• Determines which entity types to query (documents, companies, metrics, activities)

• Applies structured parameters such as recency or workflow-specific filters

• Executes retrieval against Vespa

• Iterates as needed (paginating, refining, or querying related entities)

• Assembles sufficient context before generating a response

Vespa powers this retrieval layer, enabling fast, structured queries across different object types and supporting the iterative retrieval process required by Metal’s agent-driven architecture.

Turning DDQ Chaos into Structured, Approved Intelligence

One clear example is Metal’s Due Diligence Questionnaire (DDQ) workflow. Private equity firms must respond to thousands of LP questionnaires using pre-approved answers. These responses cannot be freely generated by an LLM. They must come from content that has already been reviewed and approved by legal teams.

Answer banks change over time and are stored in unstructured formats like documents and spreadsheets. Metal indexes this data into Vespa, making the system aware of which documents are most recent. When answering a questionnaire, retrieval is prioritized not only by semantic similarity to the question but also by freshness.

This allows Metal to surface the most relevant and up-to-date approved answers, efficiently and reliably within its platform.

Scaling without Infrastructure Headaches

By building on Vespa Cloud, Metal achieved:

• Improved feature velocity: The team can introduce new entity types and workflows quickly without architectural rework

• Greater engineering focus: The team spends less time managing infrastructure and more time building differentiating product features

• Scalable retrieval architecture: Metal can onboard new clients and data volumes without redesigning retrieval.

• Confidence in long-term flexibility: Vespa is not a limiting factor as Metal expands into more advanced agent-driven workflows.

“Managing infrastructure can be a distraction. Vespa Cloud lets us focus on product.” - Sergio Prada, Co-Founder & CTO, Metal

Looking Forward: Build for an Agentic Future

Metal’s roadmap is deeply agentic. AI agents drive most interactions, deciding how best to query the platform and construct the context needed to answer sophisticated questions.

Because Vespa supports flexible, multi-entity retrieval with advanced ranking and real-time performance, Metal can:

• Expand into more advanced analysis workflows

• Build deeper relational structures between entities

• Adapt retrieval strategies dynamically as business logic evolves

The result is an institutional intelligence platform that scales in both data volume and intelligence, evolving alongside the firm it serves.

“When you’re building something ambitious, you don’t want to hit a capability wall. Vespa gives us confidence that we won’t.” - Sergio Prada, Co-Founder & CTO, Metal

RAG bridges that gap by retrieving relevant information from your data and supplying it to the model as context, so responses are grounded in real, trusted sources rather than guesswork.

The Challenge: The Quality of the Context Window

In Retrieval-Augmented Generation (RAG), the real bottleneck is the LLM’s context window. You can’t simply pass your entire dataset into a prompt—there’s a strict token budget.

Because of this, the problem isn’t just retrieving information, but retrieving the right information. When the context window is filled with loosely matched or low-quality results, the LLM has little to work with and the quality of its answers drops accordingly.

High-quality RAG depends on semantic understanding, precise retrieval, and strong ranking across diverse data types so that every token in the context window earns its place.

The Solution: Out-of-the-Box RAG on Vespa Cloud

Vespa Cloud provides an out-of-the-box Vespa RAG Blueprint designed to maximize the quality of the context sent to the LLM. Instead of relying solely on nearest-neighbor vector search, Vespa combines semantic vector retrieval with lexical BM25 scoring and applies advanced ranking, using models such as BERT, LightGBM, or custom logic—to ensure that only the strongest candidates are selected.

This hybrid retrieval and ranking approach consistently surfaces the most relevant document chunks, which significantly improves the quality of the final generated answer.

In this blog post, we’ll build a complete Retrieval-Augmented Generation (RAG) application from end to end by leveraging the OOTB Vespa RAG app on Vespa cloud. The following diagram shows the architecture we’ll be working with:

The architecture consists of two main flows: data ingestion and query processing.

Data Ingestion (one-time setup)

First, we ingest our data sources, such as documents, PDFs, or web pages by using a Python-based pipeline. The pipeline processes the data, splits it into manageable chunks, generates embeddings, and feeds everything into a Vespa Cloud RAG application that is preconfigured with a schema and ranking profiles. This step populates the search index.

Query Flow (live interaction)

1. A user enters a question in the Vespa RAG UI.

2. The UI sends the query to a Python backend, which issues a hybrid search request (combining keyword and vector retrieval) to Vespa Cloud.

3. Vespa Cloud returns the most relevant document chunks.

4. The backend sends those chunks, along with the original query, to an LLM as context.

5. The model generates an answer grounded in that context and returns it to the backend.

6. The backend streams the answer back to the UI.

This architecture ensures that generated responses are grounded in your own data, combining Vespa’s retrieval and ranking strengths with the generative capabilities of large language models.

The end-to-end setup takes about 15 minutes, plus additional time to process your documents.

Deploy Vespa RAG Blueprint to Vespa Cloud

We’ll start by deploying a preconfigured RAG Blueprint to Vespa Cloud. This gives you a high-quality retrieval stack in minutes, and it’s free to get started. All of this is done directly from the Vespa Cloud console.

Sign up for Vespa Cloud

Go to the Vespa Cloud Console and create an account. If this is your first time using Vespa Cloud, the free trial is the fastest way to get going.

Deploy RAG Blueprint

In the console, select “Deploy your first application”.

Choose “Select a sample application to deploy directly from the browser”.

Select “RAG Blueprint”.

Click “Deploy” and wait for the deployment to complete.

Save your credentials

Once deployment finishes, the console will generate an access token. Save this immediately.

That token is how Python backend authenticates with Vespa Cloud. Treat it like a password.

Continue through the remaining setup screens, then open the application view. Note your endpoint URL

In the application view you will also find the endpoint URL. Save both the endpoint URL and the token; you will need them to configure Python backend in the next section.

You can download the Vespa application package by clicking the download icon if you’d like. From there, you can start building your data feeding pipeline, frontend service UI, and more. However, this blog provides a sample end-to-end RAG application, and the same Vespa application package is included, so there’s no need to download it separately.

Behind the Scenes: What You Just Deployed

When you clicked Deploy, Vespa Cloud automatically provisioned infrastructure and deployed a complete Vespa application package. This package includes everything needed for a high-quality RAG system: schemas, indexing logic, ranking profiles, and service configuration.

In other words, you didn’t just spin up a demo, you launched a ready-to-use, high-quality retrieval engine.

Let’s take a closer look at what’s inside.

The Schema

The RAG Blueprint uses a carefully designed schema that controls how documents are stored, chunked, embedded, and retrieved:

vespa_cloud/schemas/doc.sd:

schema doc {
    document doc {
        field id type string {
            indexing: summary | attribute
        }
        field title type string {
            indexing: index | summary
            index: enable-bm25
        }
        field text type string {
        }

        # Optional metadata fields for tracking document usage
        field created_timestamp type long {
            indexing: attribute | summary
        }
        field modified_timestamp type long {
            indexing: attribute | summary
        }
        field last_opened_timestamp type long {
            indexing: attribute | summary
        }
        field open_count type int {
            indexing: attribute | summary
        }
        field favorite type bool {
            indexing: attribute | summary
        }
    }

    # Binary quantized embeddings for the title (768 floats → 96 int8)
    field title_embedding type tensor<int8>(x[96]) {
        indexing: input title | embed | pack_bits | attribute | index
        attribute {
            distance-metric: hamming
        }
    }

    # Automatically chunks text into 1024-character segments
    field chunks type array<string> {
        indexing: input text | chunk fixed-length 1024 | summary | index
        index: enable-bm25
    }

    # Binary quantized embeddings for each chunk
    field chunk_embeddings type tensor<int8>(chunk{}, x[96]) {
        indexing: input text | chunk fixed-length 1024 | embed | pack_bits | attribute | index
        attribute {
            distance-metric: hamming
        }
    }

    fieldset default {
        fields: title, chunks
    }

    document-summary top_3_chunks {
        from-disk
        summary chunks_top3 {
            source: chunks
            select-elements-by: top_3_chunk_sim_scores
        }
    }
}

What’s happening here: Your documents store their raw content in title and text. During indexing, the text field automatically split into 1024-character chunks. Embeddings are generated for both titles and chunks, then binary-quantized using pack_bits, shrinking 768 floating-point values down to just 96 int8s. This dramatically reduces storage and improves performance while still supporting efficient vector similarity search.

At the same time, BM25 is enabled for lexical matching. This combination is what enables Vespa’s hybrid retrieval: semantic matching plus exact term relevance.

Out-of-the-Box Query Profiles:

The RAG Blueprint ships with four query profiles optimized for NyRAG’s client-side RAG architecture:

NyRAG Architecture:

User Query → NyRAG (generates search queries)
          → Vespa (retrieval + ranking)
          → NyRAG (generates final answer)

Query profiles control only the Vespa retrieval/ranking step. NyRAG handles all LLM interactions.

The 4 Profiles:

1. hybrid (default, fast)
   • Retrieval: BM25 + Vector search with targetHits:100
   • Ranking: Learned linear model (logistic regression)
   • Best for: Everyday queries where you want fast, solid results
2. hybrid-with-gbdt (highest quality)
   • Retrieval: Same as hybrid (BM25 + Vector, 100 targets)
   • Ranking: Two-phase with LightGBM (GBDT) second-phase
   • Best for: Complex queries where relevance matters most (~2-3x slower)
3. deepresearch (exhaustive search)
   • Retrieval: BM25 + Vector with targetHits:10000 (100x more!)
   • Ranking: Learned linear model
   • Best for: Research scenarios needing maximum recall
4. deepresearch-with-gbdt (exhaustive + best quality)
   • Retrieval: Deep search (10k targets)
   • Ranking: Two-phase with GBDT
   • Best for: When you need both maximum recall and best ranking

For Advanced Users: Query profiles bundle complete search configurations including YQL structure (with nearestNeighbor operators), ranking profiles, and all required parameters (like learned coefficients). The Vespa application also includes rag and rag-with-gbdt profiles with searchChain=openai for server-side RAG (direct API usage), but these conflict with NyRAG’s client-side architecture and aren’t included. Learn more in the technical guide.

Which profile should you use?

• Start with hybrid for everyday use - fast and accurate
• Switch to hybrid-with-gbdt when quality matters most (harder queries)
• Use deepresearch when you need to find everything relevant (research mode)
• Try deepresearch-with-gbdt for maximum recall + quality (slowest but most thorough)

Now that your RAG Blueprint Vespa Cloud application is up and running, it’s time to add the missing pieces: a simple frontend UI and a data ingestion pipeline. For this, we’ll use NyRAG, a tool included in the RAG-app-in-15min-ragblueprint repository.

NyRAG acts as the glue for the entire RAG workflow. It reads documents from local files or websites, splits text into manageable chunks, generates embeddings, feeds everything into Vespa, and finally exposes a lightweight chat UI where you can ask questions over your data. Instead of wiring all of this together yourself, NyRAG gives you a working end-to-end system out of the box.

Install NyRAG

# Clone the repository
git clone https://github.com/vespaai-playground/RAG-app-in-15min-ragblueprint.git
cd RAG-app-in-15min-ragblueprint

# Install uv (Fast, modern Python package manager)
# macOS
brew install uv

# Linux & macOS
# curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
# powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

# Verify uv installation
uv --version

# Install dependencies using uv
uv sync
source .venv/bin/activate

# Windows (PowerShell)
# powershell -ExecutionPolicy Bypass
# . .\.venv\Scripts\activate

# Install nyrag locally
uv pip install -e .

# Verify nyrag installation
nyrag --help

Get an LLM API key

To generate final answers, NyRAG needs an OpenAI-compatible API key. The simplest way to get started is OpenRouter, which provides access to multiple LLMs through a single API.

In this walkthrough, we’ll use OpenRouter for convenience. In a real application, you’re free to swap in any compatible LLM provider. To continue, sign up for OpenRouter and generate an API key. You’ll use it in the next step when configuring NyRAG.

Start the NyRAG UI

# This script handles all configuration automatically
./run_nyrag.sh

# Windows (PowerShell)
# powershell -ExecutionPolicy Bypass
# .\run_nyrag.ps1

The run_nyrag.sh script starts the UI and wires up the configuration so NyRAG can talk to Vespa Cloud. In practice, it loads your project config, uses the token you provide for authentication, and starts the web UI on port 8000.

Open http://localhost:8000 in your browser.

Configure your project: Now you’ll configure your project using the web UI to connect to your Vespa Cloud deployment and set up document processing.

Step 1: Select and edit the example project

In the top header, the project dropdown shows “doc_example”. If you are starting from the example config, it is usually pre-selected. The configuration editor typically opens automatically; if it does not (for example you land directly in chat), open the three-dot menu (⋮) and choose “Edit Config”.

Description: Shows the project dropdown menu in the header with “doc_example” option

Note: If the configuration editor doesn’t appear (shows chat interface instead), click the three-dot menu (⋮) in the top right corner and select “Edit Config” to open it manually.

Step 2: Update your credentials

In the configuration editor, paste in the information you saved from Vespa Cloud and your LLM provider. You only need three things to get going: your Vespa tenant name, your Vespa endpoint + token, and your LLM API key.

Required fields to update:

# Your Vespa Cloud credentials (from Vespa Cloud Console)
cloud_tenant: your-tenant          # Your Vespa Cloud tenant name
vespa_cloud:
  endpoint: https://your-app.vespa-cloud.com  # Your Vespa token endpoint (not mtls)
  token: vespa_cloud_YOUR_TOKEN_HERE          # Your Vespa data plane token

# Your LLM configuration (default: OpenRouter)
llm_config:
  api_key: sk-or-v1-YOUR_KEY_HERE   # Your OpenRouter API key (or other provider)

Notes:

The default LLM provider is OpenRouter. If you switch providers, also update base_url and model to match. For the included example documents, start_loc defaults to ./dataset, so you can run the pipeline without changing anything else.

Step 3: Save and start processing

After updating the configuration, you can close the editor (changes are saved automatically) and start indexing. If you are using the example dataset, keep ./dataset as-is; otherwise, point start_loc at the folder (or site) you want to ingest. When you click “Start Indexing”, NyRAG reads your input, chunks it into 1024-character segments, generates embeddings, feeds everything to Vespa Cloud, and shows progress in the terminal panel so you can see exactly what is happening.

Description: Shows documents being processed with terminal logs displaying progress

Chat with Your Data

You can now start asking questions in the chat UI.

When you submit a query, NyRAG expands it into focused retrieval queries and sends them to Vespa. Vespa runs hybrid retrieval, combining BM25 keyword matching with vector similarity, and returns the most relevant chunks. Those chunks are packed into a compact context window and sent to the LLM, which generates an answer grounded entirely in your data.

A good way to sanity-check the setup is to start with a broad question like “What are the main topics in these documents?” and then follow up with something more specific to confirm the retrieved context makes sense.

At this point, you have a fully functional RAG application running on Vespa Cloud.

Improving Search Quality with Query Profiles

Want better search results? You can fine-tune how Vespa retrieves and ranks your documents using the Settings modal (⚙️ icon in the top right).

Change query profiles: Open the ⚙️ Settings panel, choose a Query Profile from the dropdown, and click “Save”. The very next query you run will use the new profile.

Description: Settings modal showing query profile selection dropdown with 4 available options

What each profile does:

• hybrid: Fast hybrid search (BM25 + vector) with linear ranking
• hybrid-with-gbdt: Same retrieval + advanced GBDT ranking (slower but best quality)
• deepresearch: Exhaustive search with 10,000 retrieval targets (maximum recall)
• deepresearch-with-gbdt: Exhaustive search + GBDT ranking (slowest, most thorough)

Pro tip: The quality difference between hybrid and hybrid-with-gbdt can be dramatic for complex queries. The GBDT model offers significantly better relevance at the cost of 2-3x higher latency. For research tasks where you need to find everything relevant, try deepresearch variants which cast a much wider net!

Manage Your Data

NyRAG also gives you simple tools for cleanup. Open the advanced menu (three-dot icon ⋮ in the top right) and you will find two cleanup actions. Clear Local Cache removes cached files for all projects on your machine, which is useful when you want to re-process from scratch locally. Clear Vespa Data deletes the indexed documents in Vespa for the project, which is useful when you want a clean index before re-feeding. Both actions ask for confirmation so you do not delete data by accident.

Bonus: Try Web Crawling Mode

In addition to local documents, NyRAG supports web crawling. By switching to the web_example project, you can point NyRAG at a website and have it crawl, extract, and index content automatically.

Switch to web crawling mode: Select web_example (web) from the dropdown at the top and open the configuration editor. If you are currently on the chat screen, open the three-dot menu (⋮) and choose “Edit Config” to bring the editor back. From there, update the same credential fields as you did for doc_example, then click “Start Indexing” to crawl and feed the site.

Description: Shows web crawling in progress with terminal logs displaying discovered URLs and processed pages

Web Mode Features: Web mode discovers and follows links automatically, while still respecting robots.txt and crawl delays so you do not hammer a site. It also does smart content extraction to drop navigation and boilerplate, deduplicates very similar pages, and supports resume so you can continue a crawl after interruption.

Example Use Cases: Web mode is a good fit for product documentation, knowledge bases, blog archives, help-center content, and technical wikis. In general, it works best on sites with consistent HTML structure and clean, text-heavy pages.

Tips: Start small. Crawl a limited part of a site first so you can sanity-check what gets extracted and indexed, then expand. Use exclude patterns to skip sections you do not want (for example /pricing or /sales/*), and keep an eye on the terminal output panel so you can spot loops, unexpected URLs, or pages that fail to parse.

Troubleshooting

Running into issues? We’ve got you covered! For detailed troubleshooting guides covering Vespa connection errors, LLM configuration, document processing, and more, see the Troubleshooting section in the main README.

Quick help: If you get stuck, the fastest path is usually to ask in the Vespa Slack community, where people can help you interpret logs and query behavior. If you think you found a bug or want to request an improvement, open an issue in GitHub Issues. And when you want deeper background on schema, ranking, and deployment, the Vespa Docs are your go-to reference.

Conclusion

Congratulations! You now have a working RAG app: a Vespa Cloud deployment that can retrieve high-quality context, and a small UI that lets you ingest data and chat with it.

Building a high-quality RAG system is never trivial. There are multiple moving parts to get right: the quality of the LLM, the size and management of its context window, and how effectively your retrieval system surfaces the most relevant information.

Thanks to the out-of-the-box Vespa RAG blueprint on Vespa Cloud, much of this complexity is handled for you. It comes with multiple ranking profiles, and its default hybrid retrieval setup combines vector similarity with BM25 text matching, ensuring your LLM sees the best possible context for every query.

Vespa Cloud doesn’t just make building RAG easier, it makes it scalable, fast, and reliable, giving you production-ready infrastructure, auto-scaling and observability without the headaches of self-hosting. Whether you’re experimenting with small datasets or scaling to millions of documents, Vespa Cloud provides the tools and flexibility to make your RAG project shine.

Want to dive deeper? Start with the RAG Blueprint Tutorial for a thorough conceptual walkthrough. And remember the Vespa Slack community is always there to help. Ask questions, share what you’ve built, or get advice on retrieval, ranking, and deployment strategies.

Ready to experience the power of Vespa Cloud for yourself? Sign up today and start building high-quality RAG applications with ease!

This month, we’re announcing several updates focused on retrieval quality, ranking flexibility, and developer productivity. Each feature is designed to help engineering teams build faster, more accurate, and more maintainable retrieval and ranking systems, while giving businesses better relevance, lower operational overhead, and more predictable performance at scale.

Let’s dive into what’s new.

Product updates

• Announcing the Vespa.ai Playground
• The Vespa Kubernetes Operator
• Faster result rendering with CBOR
• Pyvespa 1.0 with improved HTTP performance
• Hybrid search relevance evaluation tool
• Configurable linguistics per field
• “switch” operator in ranking expressions
• Vespa is now available on GCP Marketplace
• Feed data and run queries in the Vespa Console

Announcing the Vespa.ai Playground

The Vespa Playground is a new GitHub space where we share projects, tools, and demos built on the Vespa platform. It’s a practical place to explore real examples for embeddings, model training, and feed connectors that you can clone, run, and build on your own.

These repos are ideal for experimentation, learning, and inspiration, though they aren’t officially supported product releases.

Explore the Playground

The Vespa Kubernetes Operator

The safest, most robust and cost effective way to run Vespa is to deploy on Vespa Cloud, but for various reasons that’s not an option for everybody. For those who want to run Vespa securely at scale but can’t use Vespa Cloud we have now released the Vespa Kubernetes Operator. This brings many of the Vespa Cloud features such as security out of the box, dynamic provisioning, autoscaling and automated upgrades to your own Kubernetes environments.

Read more in the Kubernetes Operator documentation.

Faster result rendering with CBOR

Query result sets can be large, and increasingly so when the client is an LLM retrieving many chunks for model context. Layered ranking is designed to address this by extracting the most relevant content. Still, in some cases, the total latency is dominated by the time it takes to send the query response. Compressing with gzip can help, but is also CPU-intensive and slow.. From Vespa 8.623.5, json response generation is over twice as fast as before.

Another new option in this release is to use the CBOR format for query results. CBOR is a binary format so it can be serialized faster and produces smaller payloads, especially when the result contains lots of numeric data. Read more in the Query API reference and query performance guide.

Pyvespa 1.0 with improved HTTP performance

We have released the first major version of Pyvespa! This release switches the HTTP-client used by Pyvespa, from httpx to httpr, which gives big performance gains, especially for serializing and deserializing tensors, largely by taking advantage of the new CBOR serialization support in Vespa.

On preliminary benchmarks, we compared end-to-end latency for:

1. Vespa 8.591.16 + Pyvespa v0.63.0 (using JSON)

2. Vespa 8.634.24 + Pyvespa v1.0.0 (using CBOR)

The latter was ~4.9x faster when returning 400 hits with a 768-dim vector each. Performance gains will be smaller when not returning large result sets with tensors, but still significant. You may encounter different exceptions than before, but we strived to not change any user-facing API’s even if we bumped the major version.

Go to Pyvespa

Hybrid search relevance evaluation tool

Hybrid search combines lexical and embedding based search to get the best from both. One of the tasks you need to solve is to pick an embedding model that provides a good quality vs. cost tradeoff for your use case. We have done a systematic evaluation of modern alternatives in this blog.

The code used to run these experiments is now merged into Pyvespa. You can use the VespaMTEBApp to evaluate embedding model performance on any task/benchmark compatible with the mteb-library. See example usage from the tests.

Configurable linguistics per field

Vespa now lets you specify linguistics profiles on fields to select some specific linguistics processing in your Linguistics module. In Lucene Linguistics, linguistics profiles map to analyzer configuration, optionally in combination with a specific language.

For example, you can define a Lucene analyzer like this in services.xml:

  <item key="profile=whitespaceLowercase;language=en">

    <tokenizer>

      <name>whitespace</name>

    </tokenizer>

    <tokenFilters>

      <item>

        <name>lowercase</name>

      </item>

    </tokenFilters>

  </item>

And use it in the schema, under any field’s definition, like this:

field title type string {

  indexing: summary | index

  linguistics {

      profile: whitespaceLowercase

  }

}

By default the linguistics profile will be applied both when processing the text of the field and the text searching it, but you can also specify a different linguistics profile on the query side, which is useful for e.g. doing synonym query expansion.

We’ve added a sample application demonstrating how to use multiple Lucene linguistics profiles across multiple fields and updated the Vespa linguistics documentation with usage examples.

New “switch” operator in ranking expressions

We have added a “switch” function in ranking expressions as a clearer, more maintainable alternative to deeply nested if() clauses, making complex ranking easier to read, debug, and evolve.

switch (attribute(category)) {

    case "restaurant": myRestaurantFunction(),

    case "hotel": myHotelFunction(),

    default: myDefaultFunction()

}

Learn more

Vespa is now available on GCP Marketplace

Vespa Cloud is now listed on the GCP Marketplace, making it easier to deploy and manage Vespa using native Google Cloud billing and procurement. Vespa Cloud is already available on AWS Marketplace.

See details

Feed data and run queries in the Vespa Console

The onboarding experience is now even smoother for new Vespa Cloud users. When you follow the getting started guide and deploy a sample app from the browser, you can immediately feed data and run queries directly in the browser. This makes it easy to try your own data and see how it behaves in Vespa.

We also provide examples showing how to do the same using pyvespa, the Vespa CLI, or curl.

Try it Free

New content and learning resources

We published several new articles and resources since our last newsletter to help teams get more out of Vespa and stay ahead of new developments in search, RAG, and large-scale AI.

Examples and notebooks:

• playground.vespa.ai

Videos, webinars, and podcasts

• How To Scale AI in Digital Commerce Effectively
• 2025 Year in Review

Blogs and ebooks

• Clarm: Agentic AI-powered Sales for Developers with Vespa Cloud
• Embedding Tradeoffs, Quantified
• Enterprise AI Search vs. the Real Needs of Customer-Facing Apps
• Eliminating the Precision–Latency Trade-Off in Large-Scale RAG
• How Tensors Are Changing Search in Life Sciences
• The Search API Reset: Incumbents Retreat, Innovators Step Up
• Why AI Search Platforms Are Gaining Attention
• Why Life Sciences AI Is a Search Problem (Part 5 of 5)
• Why Life Sciences AI Is a Search Problem (Part 4 of 5)

Upcoming Events

Lightning Lesson: Personalized Relevance with VLMs and Sparse Vectors: February 17, 11:30am ET

• Intro to sparse vectors and tensors for efficient data handling
• Using Vision-Language Models (VLMs) to extract high quality and nuanced features from images
• Leveraging these features in sparse representations for hyper-personalized search & recommendations

Register Now

February 18: The Zero Results Problem in eCommerce

• 🔗 10am CET (EMEA): Save your spot
• 🔗 1pm ET (Americas): Save your spot

March 11: The Relevance Problem in eCommerce

• 🔗 10am CET (EMEA): Save your spot
• 🔗 1pm ET (Americas): Save your spot

March 10: Vespa Q1 Product Update

• 🔗 10am CET (EMEA): Save your spot
• 🔗 1pm ET (Americas): Save your spot

👉 Follow us on LinkedIn to stay in the loop on upcoming events, blog posts, and announcements.

Thanks for joining us in exploring the frontiers of AI with Vespa. Ready to take your projects to the next level? Deploy your application for free on Vespa Cloud today.

AI is transforming quickly. What started with Q&A chatbots has already evolved into deep research applications and, now, autonomous AI agents. Vespa is proud to be at the center of this shift, enabling some of the most proficient adopters of AI, such as Perplexity. To help organizations maximize the benefits of Vespa, we’re building a robust partner ecosystem. These partners help bring Vespa’s AI-native capabilities into real-world deployments across industries.

Meet the innovators shaping the future of AI. Today’s spotlight: Nexla

Nexla + Vespa.ai: The Power Duo for AI-Ready Data Pipelines

When AI systems fall short, it’s rarely the model’s fault. It’s the messy reality of data spread across systems and never quite staying in sync. That’s why Nexla and Vespa partnered together.

Nexla makes data usable.

Vespa makes data intelligent at scale.

Together, they turn messy, distributed enterprise data into real-time AI search, recommendation, and RAG systems, without months of custom code gluing things together.

Nexla: Making Enterprise Data Usable

Nexla is an enterprise-grade, AI-powered data integration platform that turns raw data from any source into production-ready data products. It provides a declarative, no-code way to move, transform, and validate data across ETL/ELT, reverse ETL, streaming, APIs, and RAG pipelines.

Think of Nexla as the layer that answers: “How do we reliably get the right data, in the right shape, to the systems that need it?

Core capabilities:

• 500+ Bidirectional Connectors: Pull data from databases, APIs, cloud storage, SaaS apps, and data warehouses, including systems like Salesforce, Snowflake, and Amazon S3.

• Metadata Intelligence: Nexla automatically scans sources and generates Nexsets, virtual, ready-to-use data products with schemas, samples, and validation rules. Example: If a price field suddenly switches from numeric to string, Nexla detects it before bad data reaches production search.

• Express (conversational pipelines): A conversational AI interface where you can simply describe what you need. Example: You can say, “Pull customer data from Salesforce and merge with Google Analytics,” and it builds the pipeline for you.

• Universal integration styles: Supports ELT, ETL, CDC, R-ETL, streaming, API integration, and FTP in a single platform.

Nexla processes over 1 trillion records monthly for companies like DoorDash, LinkedIn, Carrier, and LiveRamp.

Vespa: Where Retrieval Becomes Reasoning

Vespa is a production-grade AI search platform that combines a distributed text search, vector search, structured filtering, and machine-learned ranking in a single system.

Think of Vespa as the engine that answers: “Given all this data, how do we retrieve, rank, and reason over it in real time?”

It powers demanding applications like Perplexity and supports search, recommendations, personalization, and RAG at massive scale.

Core capabilities:

• Unified AI Search and Retrieval: Vespa natively combines vector and tensor search for semantic retrieval, full-text search for precise keyword matching, and structured filtering on attributes like categories, prices, and dates to enable richer, contextual search without stitching multiple systems together.

• Real-time Retrieval and Inference at Scale: Rather than separating indexing, ranking, and inference across multiple systems, Vespa performs real-time machine-learned ranking and model inference where the data lives. This means you can serve fresh, personalized results with predictable sub-100 ms latency even for large datasets.

• Multi-Phase Ranking and Custom Logic: Vespa lets you embed custom ranking logic, including ML models like XGBoost, directly into your search pipeline using ONNX. You can combine relevance signals, business rules, and semantic vectors in multi-stage ranking to fine-tune which results surface first.

• Massive Scalability with High Throughput: Designed for real-world, high-traffic applications, Vespa can scale horizontally across clusters, handling billions of documents with sub-100ms query latency and up to 100k writes per second per node.

• Multi-Vector and Multi-Modal Retrieval: Vespa natively handles multiple vectors per document, with support for token-level embeddings, ColPali-based visual document retrieval, and tensor-based computations for precise, cross-modal relevance and ranking.

GigaOm recognized Vespa as a leader in vector databases for two consecutive years, noting its performance advantages over alternatives like Elasticsearch, up to 12.9X higher throughput per CPU core for vector searches.

How Nexla and Vespa Work Together

The Nexla-Vespa partnership removes one of the hardest parts of AI systems: getting clean, well-modeled data into a high-performance retrieval engine, continuously.

Nexla recently launched a Vespa connector that makes data integration with Vespa seamless. The integration includes:

Vespa Connector in Nexla: Handles all data piping from sources like Amazon S3, PostgreSQL, Pinecone, Snowflake, and others directly into Vespa:

Vespa Nexla Plugin CLI: Automatically generates draft Vespa application packages (including schema files) directly from a Nexset, eliminating manual configuration:

This means you can move data from S3 to Vespa, migrate from Pinecone to Vespa, or sync PostgreSQL to Vespa, all without writing a single line of code.

When Nexla Clients Should Use Vespa

You’re a Nexla client. Use Vespa when you need:

Advanced AI search and RAG applications: If you’re building intelligent search, recommendation systems, or RAG applications that require hybrid search (combining semantic vector search with keyword matching and metadata filtering), Vespa is purpose-built for this. Nexla gets your data into Vespa, while Vespa delivers production-grade AI search with machine-learned ranking.

Real-time, high-scale query performance: When you need to serve thousands of queries per second across billions of documents with sub-100ms latency, Vespa’s distributed architecture scales horizontally without compromising quality. Nexla ensures your data flows continuously into Vespa with incremental updates and CDC support.

Complex ranking and inference: If your use case requires multi-phase ranking, custom ML models, or LLM integration at query time, Vespa executes these operations locally where data lives, avoiding costly data movement. Nexla prepares and transforms your data into the exact schema Vespa needs.

Cost efficiency at scale: Vespa delivers 5X infrastructure cost savings compared to alternatives like Elasticsearch while handling vector, lexical, and hybrid queries. Nexla minimizes integration costs by automating pipeline creation and schema management.

When Vespa Clients Should Use Nexla

You’re a Vespa client. Use Nexla when you need:

Multi-source data consolidation: Vespa is your search and inference engine, but data lives everywhere, S3 buckets, PostgreSQL databases, Snowflake warehouses, Salesforce CRMs, APIs, and files. Nexla connects to 500+ sources with bidirectional connectors and consolidates data into Vespa without custom ETL scripts.

Automated schema generation and management: Instead of manually writing Vespa schema files and managing schema evolution, Nexla’s Plugin CLI auto-generates schemas from your Nexsets. As source schemas change, Nexla’s metadata intelligence detects changes and propagates them downstream automatically.

Data transformation and enrichment: Before data hits Vespa, it often needs cleaning, filtering, enrichment, or format conversion. Nexla provides a no-code transformation library and supports custom SQL, Python, or JavaScript, all without maintaining separate ETL infrastructure.

Vector database migration: Moving from Pinecone, Weaviate, or another vector database to Vespa? Nexla handles the migration with zero code, extracting records, transforming data to match Vespa’s schema, and syncing documents continuously.

Data quality and monitoring: Nexla continuously monitors data flows with built-in validation rules, error handling, and automated alerts. When data quality issues arise, Nexla quarantines bad records and provides audit trails, ensuring Vespa always receives clean, trustworthy data.

Real-time and streaming pipelines: Vespa supports real-time updates, but getting real-time data from streaming sources (Kafka, APIs, databases with CDC) requires integration logic. Nexla handles streaming, batch, and hybrid integration styles, optimizing throughput and latency for each source type.

Conclusion

Nexla solves data readiness.

Vespa solves intelligence and precision at scale.

Together, they give teams a clean, practical path from raw enterprise data to real-time AI applications. Vespa gives you production-grade vector search, hybrid retrieval, and RAG capabilities at any scale. Nexla eliminates months of pipeline development and makes multi-source data flows conversational.

Ready to explore?

Start at express.dev for conversational pipeline building, or explore the Vespa connector in Nexla’s platform to see how quickly your data can power real AI applications.

Overview

Clarm helps open source software companies convert GitHub stars into revenue through AI-powered lead generation, content production, and developer support automation. When building their platform, Clarm needed a search engine that could power accurate, zero-hallucination AI responses while handling complex enrichment across millions of GitHub data points. They chose Vespa for its unified text, vector, and structured search capabilities and were able to deploy to production in under a day.

The Problem: Software / OSS Companies Struggle to Monetize

“Most OSS founders can’t get attention for their software initially. They’re so focused on building the product that marketing, SEO, and content creation get dropped. We built Clarm to automate all the growth work founders drop so they can focus on git commits,” explains Marcus Storm-Mollard, founder and CEO of Clarm.

The challenge is fundamental: 99% of successful open source is funded by businesses paying for solutions, but early-stage OSS companies lack the infrastructure to identify, engage, and convert those potential paying customers. They have thousands of GitHub stars but no clear path to revenue.

Clarm addresses this through three product pillars:

1. Lead Generation & Prospecting: The killer feature. Clarm takes repo data from customers and competitors, enriches it with signals from website visits, commits, issues, and community engagement, then ranks and identifies good-fit prospects and potential enterprise buyers.

2. Marketing & Content Production: Automated content creation from commits, PRs, and codebase analysis, helping OSS companies maintain consistent technical marketing.

3. Developer Support Automation: AI-powered support across Discord, Slack, GitHub Issues, and websites, with deep integrations and analytics for scaling customer success.

The Search Challenge

At the core of all three pillars sits a critical technical requirement: accurate, explainable search and retrieval.

“We realized early that search, not generation, was the real problem to solve. Generating LLM answers isn’t hard. Finding the right information to base them on is everything,” Marcus notes.

Clarm needed a search engine that could:

• Handle hybrid retrieval (combining text search, vector embeddings, and structured filters)
• Power zero-hallucination AI responses grounded in verifiable context
• Process and rank millions of GitHub data points in real-time
• Support complex multi-signal enrichment for lead scoring
• Scale cost-effectively on a startup budget

Traditional vector databases like Supabase or search engines like Elasticsearch couldn’t deliver the unified, production-grade retrieval required for Clarm’s zero-hallucination architecture.

The Solution: Vespa’s Production-Grade Hybrid Search

Marcus discovered Vespa after researching how companies like Perplexity and Onyx built their advanced retrieval systems.

“We really liked that Vespa started as a search engine and evolved into a vector-based system. It made so much sense for what we were building. Vespa’s ranking and tensoring are built in, so we know our results are accurate and relevant right out of the box,” Marcus explains.

Rapid Deployment: Less Than One Day to Production

Clarm began experimenting with Vespa’s Docker image for local development, then transitioned to Vespa Cloud for production deployment during their Y Combinator batch.

“It took about half a day to set up how we wanted it. That speed of onboarding made a huge impact during YC. We just deployed it, and it worked,” Marcus recalls.

The quick deployment was critical. Clarm was racing toward Demo Day and couldn’t afford weeks of infrastructure setup. Vespa’s unified approach eliminated the complexity of stitching together multiple systems for text, vector, and structured search.

Key Vespa Capabilities Powering Clarm

• Unified Retrieval Pipeline
  • Single query endpoint combining text search, vector similarity, and structured filters - no need to orchestrate multiple databases or services.
• Built-in Ranking & Tensor Operations
  • Native support for complex ranking models and tensor operations means Clarm can implement sophisticated lead scoring without custom ranking layers.
• Real-Time Indexing
  • GitHub events, user interactions, and enrichment signals are instantly searchable, enabling live lead intelligence and up-to-date AI responses.
• Scalable Cloud Deployment
  • Automatic scaling and high availability handled by Vespa Cloud, allowing Clarm’s two-person engineering team to focus on product features instead of infrastructure operations.
• Developer-Friendly Architecture
  • Docker-based local development, straightforward schema design, and comprehensive documentation enabled rapid prototyping and iteration.

The Results

Clarm’s decision to build on Vespa Cloud delivered immediate impact:

• <1 Day to Production: From prototype to live search infrastructure deployed during YC
• Zero-Hallucination Architecture: Accurate retrieval enabling trustworthy AI responses grounded in verifiable data
• High-Quality Lead Intelligence: Sophisticated ranking of GitHub data points across 50K+ collective stars from customers like Better Auth (23.3K stars) and Cua (11.3K stars)
• Exceptional Support: Direct collaboration with Vespa’s engineering team throughout development

“The setup was easy, the support from the Vespa team was incredible, and everything just worked. We didn’t need to look anywhere else,” Marcus emphasizes.

Customer Success: GitHub Stars Becoming Revenue

Clarm’s customers are seeing measurable results from the AI-powered lead generation platform:

• Better Auth: Grew from 8K to 23.3K GitHub stars in 3 months with Clarm’s lead gen and engagement automation
• c/ua: Scaled from 5K to 11.3K stars while identifying and converting enterprise prospects
• Skyvern AI: after struggling with after hitting 19k stars, reduced support workload by 94% with Clarm across Github, Discord, and Slack
• Engagement Depth: Developers “pair programming” with Clarm’s AI agents for extended sessions, sending thousands of queries a day and sessions lasting up to 22 hours

What’s Next: Building the Future of OSS Monetization

Clarm represents a new category of growth infrastructure built specifically for software and open source companies. By combining Vespa’s production-grade retrieval with their own zero-hallucination agent framework, Clarm is proving that AI-powered sales and marketing can be trustworthy, explainable, and grounded in truth.

“We’re focused on proving product value and retaining customers right now. Everything depends on us growing our customers’ MRR and showing software and OSS companies they can build sustainable businesses,” Marcus shares.

That focus is reflected in Clarm’s positioning: “You build awesome software. Now build a business.” It resonates with software founders who want to monetize without compromising their community values. By recognizing that a vast majority of successful open source is ultimately funded by businesses paying for solutions, Clarm offers a clear path forward: free software for the community, paid solutions for enterprises.

Conclusion

Clarm’s architecture reinforces a lesson many teams learn the hard way: LLMs are only as reliable as the retrieval systems behind them. By treating retrieval as a first-class system, built on Vespa Cloud, Clarm unified text search, vector similarity, structured filtering, and ranking into a single production-grade platform, eliminating the fragility and guesswork common in vector-only stacks.

The result is an agentic AI platform that can reason over live data, explain its outputs, and scale predictably without stitching together multiple databases or post-hoc ranking layers. This foundation enabled a small team to move from prototype to production in days, operate across millions of GitHub signals, and help open source companies turn community adoption into sustainable revenue.

More importantly, Clarm’s success offers a blueprint for any organization building serious AI applications: when retrieval is reliable, ranking is expressive, and data is always fresh, AI systems become trustworthy enough to power real business outcomes. Clarm is building the future of OSS monetization, and Vespa is the retrieval engine making it possible.

The typical approach: open the MTEB leaderboard, find the “Retrieval” column, sort descending, pick something that fits your size budget. Done, right?

Not quite. MTEB doesn’t tell you:

• How fast is inference on your actual hardware?
• What happens when you quantize the model weights?
• How much quality do you lose with binary vectors?
• Does this model even work well in a hybrid setup?

So we ran the experiments ourselves. We picked models from the MTEB Retrieval leaderboard with these criteria:

• Under 500M parameters (practical for most deployments)
• Open license
• ONNX weights available (required for Vespa)
• At least 10k downloads in the last month (actually used in production)

For each model, we benchmarked across:

• Model quantizations (FP32, FP16, INT8)
• Vector precisions (float, bfloat16, binary)
• Matryoshka dimensions (for models that support it)
• Real hardware (Graviton3, Graviton4, T4 GPU)
• Hybrid retrieval (semantic, RRF, and score normalization methods)

Spoiler: We found some really attractive tradeoffs - 32x memory reduction, 4x faster inference, with nearly identical quality.

What MTEB doesn’t show you

Model quantization

Vespa uses ONNX runtime for embedding inference. Most models on HuggingFace ship with multiple ONNX variants - here’s Alibaba-NLP/gte-modernbert-base as an example:

Lower precision weights = smaller model = faster inference. But how much faster, and what’s the quality hit?

• On CPU: INT8 models run 2.7-3.4x faster while keeping 94-98% of the quality
• On GPU: INT8 is actually 4-5x slower than FP32. Don’t do this.

The difference between 30ms and 100ms query latency is huge. If you’re on CPU, INT8 is often a no-brainer.

On GPU, use FP16 instead - you get ~2x speedup with no meaningful quality loss.

GPU vs CPU: The T4 GPU runs 4-7x faster than Graviton3 for embedding inference. If you’re processing high query volumes or doing bulk indexing, GPU may be worth it.

Vector precision

Model quantization affects inference speed. Vector precision affects storage and search speed. Different knobs, both important.

Here’s the math for 100 million 768-dimensional embeddings:

That’s a 32x difference between FP32 and binary. When memory is what forces you to add more nodes, this matters a lot.

bfloat16 is free: In our benchmarks, bfloat16 vectors show zero quality loss compared to FP32 - it’s a 2x storage reduction you can take without any tradeoff.

Matryoshka dimensions

Some models support Matryoshka Representation Learning (MRL) - you can truncate the embedding to fewer dimensions and still get decent results. Fewer dimensions = less storage, faster search.

Here’s EmbeddingGemma at different dimension sizes:

Source: EmbeddingGemma paper

Interestingly, EmbeddingGemma actually scores higher at 512 dimensions than at 768. We didn’t dig into why - it may be an artifact of the smaller evaluation set - but it’s a reminder that more dimensions isn’t always better.

Not all models support this - check the model card before truncating. If it wasn’t trained for MRL, slicing dimensions will tank your quality.

Inference speed

If you have a 200ms latency budget and your embedding model takes 150ms, you’re in trouble. We benchmarked actual inference times so you can plan accordingly.

We measured two things for each model:

1. Query latency - how long to embed an 8-word query
2. Document throughput - embeddings per second for 103-word docs

Tested on three AWS instance types:

• c7g.2xlarge - Graviton 3 (ARM CPU)
• g4dn.xlarge - T4 GPU
• m8g.xlarge - Graviton 4 (ARM CPU)

These numbers are pure ONNX inference time. Your actual indexing throughput will also depend on HNSW config and existing index size, but embedding inference is usually the bottleneck.

Quality

We evaluated all models on NanoBEIR, a smaller but representative subset of the BEIR benchmark. This let us run a lot of experiments without waiting forever.

For each model, we measured nDCG@10 across four retrieval strategies:

• Semantic only - pure vector similarity
• RRF (Reciprocal Rank Fusion) - combines BM25 and vector rankings
• Atan hybrid - normalizes scores using arctangent before combining
• Linear hybrid - linear normalization before combining

The hybrid methods consistently outperform pure semantic search. Every single model in our benchmark scored higher with hybrid retrieval than semantic-only. On average, the best hybrid method beats semantic-only by 3-5 percentage points. That’s a meaningful lift you get “for free” by just using BM25 alongside your vectors.

We also tested each model with binarized vectors (int8). This is where things get interesting:

• ModernBERT models barely flinch - Alibaba GTE ModernBERT retains 98% of quality (0.670 binary vs 0.685 float)
• E5 models take a bigger hit - E5-base-v2 drops to 92% (0.602 binary vs 0.651 float), and E5-small-v2 to just 87%

The takeaway: not all models are created equal for binary quantization. The newer ModernBERT-based models handle it much better than the E5 family. Make sure to check before assuming you can just binarize everything.

Interactive leaderboard

We built an interactive leaderboard so you can explore the full results yourself. Filter by hardware, sort by different metrics, and expand each model to see the full breakdown across dimensions and precisions. Open in full screen.

Getting started with Vespa

Ready to put this into practice? Here’s how to configure an embedding model in Vespa:

<component id="alibaba_gte_modernbert_int8" type="hugging-face-embedder">
    <transformer-model model-id="alibaba-gte-modernbert"/>
    <max-tokens>8192</max-tokens>
    <pooling-strategy>cls</pooling-strategy>
</component>

Here’s a schema with a binarized embedding field (96 dimensions = 768 bits packed):

schema doc {
  document doc {
    field id type string {
      indexing: summary | attribute
    }
    field text type string {
      indexing: index | summary
      index: enable-bm25
    }
  }
  field embedding_alibaba_gte_modernbert_int8_96_int8 type tensor<int8>(x[96]) {
    indexing: input text | embed alibaba_gte_modernbert_int8 | pack_bits | index | attribute
    attribute {
      distance-metric: hamming
    }
    index {
      hnsw {
        max-links-per-node: 16
        neighbors-to-explore-at-insert: 200
      }
    }
  }
}

And a rank profile using linear normalization for hybrid scoring:

rank-profile hybrid_linear {
  inputs {
    query(q) tensor<int8>(x[96])
  }
  function similarity() {
    expression {
      1 - (distance(field, embedding_alibaba_gte_modernbert_int8_96_int8) / 768)
    }
  }
  first-phase {
    expression: similarity
  }
  global-phase {
    expression: normalize_linear(bm25(text)) + normalize_linear(similarity)
    rerank-count: 1000
  }
  match-features {
    similarity
    bm25(text)
  }
}

Check out the embedding documentation for full details on configuration, including how to set up binary quantization and hybrid search.

Going further

Binary vectors are fast - really fast. Vespa can do ~1 billion hamming distance calculations per second, roughly 7x more than prenormalized angular distance. That speed difference means you can crank up targetHits significantly and still stay within latency budget. More candidates evaluated = better recall. So binary vectors aren’t just about 32x storage savings - they give you headroom to tune for quality too.

And luckily, Vespa’s phased ranking architecture lets you make up for any remaining quality loss in later phases. You can retrieve candidates with hamming distance, then rescore in any of the following ways:

• float-binary - Use float for query vector, and unpack the bits of document vector to float for angular distance calculation. Example
• float-float - Retrieve with hamming distance but rerank with full-precision vectors paged in from disk. Should be limited to a small candidate set.
• int8-int8 - Same as float-float, with int8 vectors (scalar quantization, not to be confused with binary quantization) for both query and document. Faster and more storage-efficient than float-float, with a small precision cost.

See this great huggingface blog post for more details on these techniques.

For even better results, add a cross-encoder reranker as a final stage. Or (especially if you have several user signals or features), train a GBDT model to learn optimal combinations.

The beauty of Vespa’s ranking expressions is that you can mix and match all of these - BM25, a bunch of other built-in features, vectors, rerankers, learned models - however you want.

A few caveats

Multilingual support

If you need to support multiple languages, your options narrow. The multilingual-e5-base model handles 100+ languages but comes with a quality tradeoff compared to English-only models. For English-only workloads, stick with the specialized models.

Context length

Document length matters too. Many newer models handle 8192 tokens, EmbeddingGemma can take 2048, while the E5 family tops out at 512. If your documents are long, look at benchmarks like LoCo (Long Document Retrieval) - NanoBEIR won’t tell you much here.

For long documents, check out Vespa’s layered ranking - it lets you rank chunks within documents so you’re not forced to return irrelevant chunks from top-ranking docs.

Test on your own data

NanoBEIR is a good starting point, but your domain matters. A model that tops the leaderboard on scientific papers might struggle with product descriptions, legal documents, or your internal knowledge base.

Benchmark rankings can be misleading for specialized domains. The models we tested were trained on general web data - if your corpus looks very different (medical records, source code, niche industry jargon), the relative rankings might shuffle significantly.

We’ve open-sourced the benchmarking code in pyvespa so you can run the same experiments on any model with any dataset compatible with the MTEB library. Swap in your own data and see how different models actually perform for your use case.

Consider finetuning

If off-the-shelf models underperform on your domain, finetuning can help significantly. Even a small set of query-document pairs from your actual data can boost relevance.

Tools like sentence-transformers make this straightforward. The ROI is often worth it for production systems where a few percentage points of nDCG translate to real user impact.

Wrapping up

The “best” embedding model depends entirely on your constraints. But now you have real data to make that call:

• Cost sensitive? Binary quantization with a compatible model (like GTE ModernBERT) gives you 32x savings with minimal quality loss.
• Running on CPU? INT8 model quantization speeds up inference 2.7-3.4x.
• Need great quality? Alibaba GTE ModernBERT + hybrid search is hard to beat.
• Latency-critical? E5-small-v2 with INT8 can do a query inference in only 2.5ms on Graviton3.

The interactive leaderboard above has all the details. Explore, filter, and find the sweet spot for your use case.

For those interested in learning more about Vespa, join the Vespa community on Slack to exchange ideas, seek assistance from the community, or stay in the loop on the latest Vespa developments.

Originally posted 25th Aug. 2025 on The New Stack

In my years working across life sciences, one question comes up again and again: What’s next for AI in our field? The truth is that the life sciences industry faces challenges unlike any other.

Where a bank or retailer might deploy AI chatbots to improve customer service, our world is defined by enormous, messy datasets, including clinical trials, lab results, publications and patient records. This must be interpreted with care. The stakes are not just efficiency or convenience; they are breakthroughs in treatment, safety and patient outcomes.

That’s why I believe the real opportunity for generative AI (GenAI) in the life sciences is not in chatbots, but in enabling deep and precise retrieval. Success here means connecting across multiple sources, reconciling heterogeneous data and surfacing insights that a human researcher would struggle to piece together.

Imagine asking: “Find me colorectal cancer trials using ZALTRAP [a drug] with the most recent supporting publications.” GenAI, when applied effectively, can handle that complexity, and this is where the next frontier begins.

From Traditional Search to AI-Driven Discovery

For decades, search in life sciences has mostly meant keyword lookups or rule-based retrieval. Researchers, clinicians and pharma teams relied on these tools to sift through scientific literature, clinical trial data, patents and regulatory filings. They worked well enough for simple, well-defined questions. But as soon as you needed to account for domain-specific language, synonyms or the complex relationships between diseases, molecules and pathways, traditional search hit its limits.

The result? Endless manual refinements, stitching insights together from different sources and lots of time spent just finding the right information.

Now, with GenAI and large language models (LLMs), that’s changing. LLM-powered search understands meaning, not just exact words. You can ask complex, natural-language questions and get results that connect the dots across literature, trials and patents — even when they use different terminology. This opens up entirely new ways of working: identifying drug repurposing opportunities hidden in disconnected studies, accelerating biomarker discovery or finding previously unseen links between biological entities. It’s faster, more comprehensive and far less manual.

Why Tensors Matter in This Shift

Life sciences data comes in all shapes and sizes — omics data, 3D protein structures, medical images, regulatory documents, clinical trial reports and more. Most of it is unstructured or semi-structured, which makes it tricky for AI systems to find and assemble relevant information quickly. Given the nature of life sciences, accuracy is critical. “Good enough” seldom exists.

This is where tensors come in.

So, what is a tensor? Think of it as a multidimensional data container. A vector is a one-dimensional list of numbers. A matrix is two-dimensional. A tensor goes beyond that, capturing multiple dimensions at once. This allows AI models to represent complex relationships — like spatial configurations of proteins or contextual relationships between words in a scientific article — even if those pieces of information are far apart.

In other words, tensors let AI “see” and learn patterns that are deeply embedded across different dimensions of data.

Tensors in Action: Protein Structures

Take structural biology as an example. Models like AlphaFold use 3D tensors to represent the spatial relationships between amino acids. These tensors allow the model to learn how proteins fold, twist and interact — crucial knowledge for understanding disease mechanisms and designing new therapies.

When you embed a protein as a tensor, you preserve:

• Sequential data (the order of amino acids)
• Spatial relationships (how parts of the protein fold and connect)
• Biochemical properties (like charge or hydrophobicity)

This rich representation lets machine learning (ML) models predict protein folding, identify binding sites, map protein-protein interactions and even discover new drug targets.

The same idea applies beyond proteins.

Medical imaging, for example, can use tensors to encode not just pixels, but also their contextual relevance, helping AI detect subtle cancer markers even in noisy scans. In clinical settings, tensors help AI analyze data streams from wearables or Internet of Things (IoT) devices in real time, enabling faster interventions.

Beyond Retrieval: AI Agents in Life Sciences

AI agents are another emerging application. Think of them as intelligent assistants that continuously gather, analyze and synthesize information across fragmented data sources. An AI agent could monitor new literature, clinical trials and regulatory updates in real time, summarize findings and even suggest next research steps.

Good agents don’t just fetch information — they connect it, building context and reason through problems step by step.

The key here is multistep reasoning. Good agents don’t just fetch information — they connect it, building context and reason through problems step by step.

This means faster reasoning, better accuracy and more meaningful insights. This allows you to stitch together multimodal data and ask questions across modalities and time. For example, as illustrated in the example below, you can now find patients for trial recruitment for a disease subtype based on certain progression (or regression) images over time. You can do this by combining into a tensor the patient’s medical record, biomarker assays, histopathology slides and any other prognosis outcome notes.

Why This Matters

Life sciences are moving into an era where data is simply too complex and too vast for traditional tools. Tensors provide the foundation for AI models to handle this complexity, enabling everything from better search to advanced reasoning. Whether it’s predicting protein structures, extracting insights from clinical data or powering AI agents that help researchers focus on discovery rather than data wrangling, tensors are quietly becoming the backbone of the next wave of AI in life sciences.

Originally posted 7th Nov. 2025 on The New Stack

The search landscape is shifting. In recent months, Microsoft announced the retirement of the Bing Search API, while Google limited its own API to a maximum of 10 results per query. These moves mark a notable change in the way the web’s dominant search providers view access to their data and who gets to build on it.

For more than a decade, search APIs like Bing and Google Custom Search have been part of the web’s plumbing. Developers have used them to retrieve web results, images and news without maintaining their own indexes. Enterprises have embedded them in applications such as customer support, knowledge bases and market intelligence to provide external context. Startups and research teams have used them to collect training data, ground language models or perform competitive analysis without running their own crawlers.

In short, search APIs have offered a simple way to access the open web programmatically, bridging the gap between consumer search and enterprise information retrieval.

The AI Shift

The emergence of generative AI has changed what the search infrastructure needs to deliver. With retrieval-augmented generation (RAG) becoming central to AI systems, developers now require flexible retrieval layers within the AI pipeline, not just returning links.

Against this backdrop, the timing of Microsoft and Google’s decisions stands out. Microsoft has folded search access into Azure’s AI stack through its Grounding with Bing Search feature for AI agents, while Google continues to reduce external visibility into its own results. Limiting queries to 10 results per call fits with its long-standing goal of minimizing bulk data extraction and automated scraping.

The business thinking is clear: Both companies are steering developers away from large-scale, open retrieval and toward AI-mediated access inside their own ecosystems. Full result sets are expensive to serve and often used by automated systems such as SEO platforms, data-mining tools or research crawlers rather than by interactive users. Restricting APIs helps contain those costs while repositioning web data as a controlled resource for higher-level AI services.

A Reset, Not a Retreat

This isn’t a collapse of search, but a realignment of control. The open, list-based APIs of the past belong to an era where raw results were the product. In the generative AI era, incumbents are redefining search around answers, grounding and context, tightly coupled with their cloud ecosystems.

But as the large providers step back, new players are moving in. Perplexity and Parallel represent a new generation of search APIs designed for AI workloads. They publish benchmarks, expose APIs openly and emphasize retrieval quality and low latency, the performance characteristics that matter most in RAG and agentic systems. You can read more about the Perplexity search API here.

Perplexity has also shown that it outperforms Google on relevance for RAG-style tasks. Not to be outdone, Parallel, founded by Twitter’s former CEO, Parag Agrawal, recently reported better results than Perplexity, using Perplexity’s own evaluation tool.

A Hot Market, New Foundations

The search API market is heating up again, this time around AI native infrastructure. Beneath Perplexity and Parallel is a common component: Vespa, the open source engine built for large-scale retrieval, ranking and machine learning inference.

Vespa’s role in these systems reflects a broader shift in architecture: Search infrastructure is now part of the AI stack itself. As models depend more on retrieval, factors such as performance, scalability and the ability to combine structured and unstructured data have become key differentiators.

The incumbents are narrowing access; the innovators are expanding it. Either way, search is once again at the center of how the web is organized, only this time, it’s being rebuilt for AI.