RAG bridges that gap by retrieving relevant information from your data and supplying it to the model as context, so responses are grounded in real, trusted sources rather than guesswork.

The Challenge: The Quality of the Context Window

In Retrieval-Augmented Generation (RAG), the real bottleneck is the LLM’s context window. You can’t simply pass your entire dataset into a prompt—there’s a strict token budget.

Because of this, the problem isn’t just retrieving information, but retrieving the right information. When the context window is filled with loosely matched or low-quality results, the LLM has little to work with and the quality of its answers drops accordingly.

High-quality RAG depends on semantic understanding, precise retrieval, and strong ranking across diverse data types so that every token in the context window earns its place.

The Solution: Out-of-the-Box RAG on Vespa Cloud

Vespa Cloud provides an out-of-the-box Vespa RAG Blueprint designed to maximize the quality of the context sent to the LLM. Instead of relying solely on nearest-neighbor vector search, Vespa combines semantic vector retrieval with lexical BM25 scoring and applies advanced ranking, using models such as BERT, LightGBM, or custom logic—to ensure that only the strongest candidates are selected.

This hybrid retrieval and ranking approach consistently surfaces the most relevant document chunks, which significantly improves the quality of the final generated answer.

In this blog post, we’ll build a complete Retrieval-Augmented Generation (RAG) application from end to end by leveraging the OOTB Vespa RAG app on Vespa cloud. The following diagram shows the architecture we’ll be working with:

The architecture consists of two main flows: data ingestion and query processing.

Data Ingestion (one-time setup)

First, we ingest our data sources, such as documents, PDFs, or web pages by using a Python-based pipeline. The pipeline processes the data, splits it into manageable chunks, generates embeddings, and feeds everything into a Vespa Cloud RAG application that is preconfigured with a schema and ranking profiles. This step populates the search index.

Query Flow (live interaction)

1. A user enters a question in the Vespa RAG UI.

2. The UI sends the query to a Python backend, which issues a hybrid search request (combining keyword and vector retrieval) to Vespa Cloud.

3. Vespa Cloud returns the most relevant document chunks.

4. The backend sends those chunks, along with the original query, to an LLM as context.

5. The model generates an answer grounded in that context and returns it to the backend.

6. The backend streams the answer back to the UI.

This architecture ensures that generated responses are grounded in your own data, combining Vespa’s retrieval and ranking strengths with the generative capabilities of large language models.

The end-to-end setup takes about 15 minutes, plus additional time to process your documents.

Deploy Vespa RAG Blueprint to Vespa Cloud

We’ll start by deploying a preconfigured RAG Blueprint to Vespa Cloud. This gives you a high-quality retrieval stack in minutes, and it’s free to get started. All of this is done directly from the Vespa Cloud console.

Sign up for Vespa Cloud

Go to the Vespa Cloud Console and create an account. If this is your first time using Vespa Cloud, the free trial is the fastest way to get going.

Deploy RAG Blueprint

In the console, select “Deploy your first application”.

Choose “Select a sample application to deploy directly from the browser”.

Select “RAG Blueprint”.

Click “Deploy” and wait for the deployment to complete.

Save your credentials

Once deployment finishes, the console will generate an access token. Save this immediately.

That token is how Python backend authenticates with Vespa Cloud. Treat it like a password.

Continue through the remaining setup screens, then open the application view. Note your endpoint URL

In the application view you will also find the endpoint URL. Save both the endpoint URL and the token; you will need them to configure Python backend in the next section.

You can download the Vespa application package by clicking the download icon if you’d like. From there, you can start building your data feeding pipeline, frontend service UI, and more. However, this blog provides a sample end-to-end RAG application, and the same Vespa application package is included, so there’s no need to download it separately.

Behind the Scenes: What You Just Deployed

When you clicked Deploy, Vespa Cloud automatically provisioned infrastructure and deployed a complete Vespa application package. This package includes everything needed for a high-quality RAG system: schemas, indexing logic, ranking profiles, and service configuration.

In other words, you didn’t just spin up a demo, you launched a ready-to-use, high-quality retrieval engine.

Let’s take a closer look at what’s inside.

The Schema

The RAG Blueprint uses a carefully designed schema that controls how documents are stored, chunked, embedded, and retrieved:

vespa_cloud/schemas/doc.sd:

schema doc {
    document doc {
        field id type string {
            indexing: summary | attribute
        }
        field title type string {
            indexing: index | summary
            index: enable-bm25
        }
        field text type string {
        }

        # Optional metadata fields for tracking document usage
        field created_timestamp type long {
            indexing: attribute | summary
        }
        field modified_timestamp type long {
            indexing: attribute | summary
        }
        field last_opened_timestamp type long {
            indexing: attribute | summary
        }
        field open_count type int {
            indexing: attribute | summary
        }
        field favorite type bool {
            indexing: attribute | summary
        }
    }

    # Binary quantized embeddings for the title (768 floats → 96 int8)
    field title_embedding type tensor<int8>(x[96]) {
        indexing: input title | embed | pack_bits | attribute | index
        attribute {
            distance-metric: hamming
        }
    }

    # Automatically chunks text into 1024-character segments
    field chunks type array<string> {
        indexing: input text | chunk fixed-length 1024 | summary | index
        index: enable-bm25
    }

    # Binary quantized embeddings for each chunk
    field chunk_embeddings type tensor<int8>(chunk{}, x[96]) {
        indexing: input text | chunk fixed-length 1024 | embed | pack_bits | attribute | index
        attribute {
            distance-metric: hamming
        }
    }

    fieldset default {
        fields: title, chunks
    }

    document-summary top_3_chunks {
        from-disk
        summary chunks_top3 {
            source: chunks
            select-elements-by: top_3_chunk_sim_scores
        }
    }
}

What’s happening here: Your documents store their raw content in title and text. During indexing, the text field automatically split into 1024-character chunks. Embeddings are generated for both titles and chunks, then binary-quantized using pack_bits, shrinking 768 floating-point values down to just 96 int8s. This dramatically reduces storage and improves performance while still supporting efficient vector similarity search.

At the same time, BM25 is enabled for lexical matching. This combination is what enables Vespa’s hybrid retrieval: semantic matching plus exact term relevance.

Out-of-the-Box Query Profiles:

The RAG Blueprint ships with four query profiles optimized for NyRAG’s client-side RAG architecture:

NyRAG Architecture:

User Query → NyRAG (generates search queries)
          → Vespa (retrieval + ranking)
          → NyRAG (generates final answer)

Query profiles control only the Vespa retrieval/ranking step. NyRAG handles all LLM interactions.

The 4 Profiles:

1. hybrid (default, fast)
   • Retrieval: BM25 + Vector search with targetHits:100
   • Ranking: Learned linear model (logistic regression)
   • Best for: Everyday queries where you want fast, solid results
2. hybrid-with-gbdt (highest quality)
   • Retrieval: Same as hybrid (BM25 + Vector, 100 targets)
   • Ranking: Two-phase with LightGBM (GBDT) second-phase
   • Best for: Complex queries where relevance matters most (~2-3x slower)
3. deepresearch (exhaustive search)
   • Retrieval: BM25 + Vector with targetHits:10000 (100x more!)
   • Ranking: Learned linear model
   • Best for: Research scenarios needing maximum recall
4. deepresearch-with-gbdt (exhaustive + best quality)
   • Retrieval: Deep search (10k targets)
   • Ranking: Two-phase with GBDT
   • Best for: When you need both maximum recall and best ranking

For Advanced Users: Query profiles bundle complete search configurations including YQL structure (with nearestNeighbor operators), ranking profiles, and all required parameters (like learned coefficients). The Vespa application also includes rag and rag-with-gbdt profiles with searchChain=openai for server-side RAG (direct API usage), but these conflict with NyRAG’s client-side architecture and aren’t included. Learn more in the technical guide.

Which profile should you use?

• Start with hybrid for everyday use - fast and accurate
• Switch to hybrid-with-gbdt when quality matters most (harder queries)
• Use deepresearch when you need to find everything relevant (research mode)
• Try deepresearch-with-gbdt for maximum recall + quality (slowest but most thorough)

Now that your RAG Blueprint Vespa Cloud application is up and running, it’s time to add the missing pieces: a simple frontend UI and a data ingestion pipeline. For this, we’ll use NyRAG, a tool included in the RAG-app-in-15min-ragblueprint repository.

NyRAG acts as the glue for the entire RAG workflow. It reads documents from local files or websites, splits text into manageable chunks, generates embeddings, feeds everything into Vespa, and finally exposes a lightweight chat UI where you can ask questions over your data. Instead of wiring all of this together yourself, NyRAG gives you a working end-to-end system out of the box.

Install NyRAG

# Clone the repository
git clone https://github.com/vespaai-playground/RAG-app-in-15min-ragblueprint.git
cd RAG-app-in-15min-ragblueprint

# Install uv (Fast, modern Python package manager)
# macOS
brew install uv

# Linux & macOS
# curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
# powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

# Verify uv installation
uv --version

# Install dependencies using uv
uv sync
source .venv/bin/activate

# Windows (PowerShell)
# powershell -ExecutionPolicy Bypass
# . .\.venv\Scripts\activate

# Install nyrag locally
uv pip install -e .

# Verify nyrag installation
nyrag --help

Get an LLM API key

To generate final answers, NyRAG needs an OpenAI-compatible API key. The simplest way to get started is OpenRouter, which provides access to multiple LLMs through a single API.

In this walkthrough, we’ll use OpenRouter for convenience. In a real application, you’re free to swap in any compatible LLM provider. To continue, sign up for OpenRouter and generate an API key. You’ll use it in the next step when configuring NyRAG.

Start the NyRAG UI

# This script handles all configuration automatically
./run_nyrag.sh

# Windows (PowerShell)
# powershell -ExecutionPolicy Bypass
# .\run_nyrag.ps1

The run_nyrag.sh script starts the UI and wires up the configuration so NyRAG can talk to Vespa Cloud. In practice, it loads your project config, uses the token you provide for authentication, and starts the web UI on port 8000.

Open http://localhost:8000 in your browser.

Configure your project: Now you’ll configure your project using the web UI to connect to your Vespa Cloud deployment and set up document processing.

Step 1: Select and edit the example project

In the top header, the project dropdown shows “doc_example”. If you are starting from the example config, it is usually pre-selected. The configuration editor typically opens automatically; if it does not (for example you land directly in chat), open the three-dot menu (⋮) and choose “Edit Config”.

Description: Shows the project dropdown menu in the header with “doc_example” option

Note: If the configuration editor doesn’t appear (shows chat interface instead), click the three-dot menu (⋮) in the top right corner and select “Edit Config” to open it manually.

Step 2: Update your credentials

In the configuration editor, paste in the information you saved from Vespa Cloud and your LLM provider. You only need three things to get going: your Vespa tenant name, your Vespa endpoint + token, and your LLM API key.

Required fields to update:

# Your Vespa Cloud credentials (from Vespa Cloud Console)
cloud_tenant: your-tenant          # Your Vespa Cloud tenant name
vespa_cloud:
  endpoint: https://your-app.vespa-cloud.com  # Your Vespa token endpoint (not mtls)
  token: vespa_cloud_YOUR_TOKEN_HERE          # Your Vespa data plane token

# Your LLM configuration (default: OpenRouter)
llm_config:
  api_key: sk-or-v1-YOUR_KEY_HERE   # Your OpenRouter API key (or other provider)

Notes:

The default LLM provider is OpenRouter. If you switch providers, also update base_url and model to match. For the included example documents, start_loc defaults to ./dataset, so you can run the pipeline without changing anything else.

Step 3: Save and start processing

After updating the configuration, you can close the editor (changes are saved automatically) and start indexing. If you are using the example dataset, keep ./dataset as-is; otherwise, point start_loc at the folder (or site) you want to ingest. When you click “Start Indexing”, NyRAG reads your input, chunks it into 1024-character segments, generates embeddings, feeds everything to Vespa Cloud, and shows progress in the terminal panel so you can see exactly what is happening.

Description: Shows documents being processed with terminal logs displaying progress

Chat with Your Data

You can now start asking questions in the chat UI.

When you submit a query, NyRAG expands it into focused retrieval queries and sends them to Vespa. Vespa runs hybrid retrieval, combining BM25 keyword matching with vector similarity, and returns the most relevant chunks. Those chunks are packed into a compact context window and sent to the LLM, which generates an answer grounded entirely in your data.

A good way to sanity-check the setup is to start with a broad question like “What are the main topics in these documents?” and then follow up with something more specific to confirm the retrieved context makes sense.

At this point, you have a fully functional RAG application running on Vespa Cloud.

Improving Search Quality with Query Profiles

Want better search results? You can fine-tune how Vespa retrieves and ranks your documents using the Settings modal (⚙️ icon in the top right).

Change query profiles: Open the ⚙️ Settings panel, choose a Query Profile from the dropdown, and click “Save”. The very next query you run will use the new profile.

Description: Settings modal showing query profile selection dropdown with 4 available options

What each profile does:

• hybrid: Fast hybrid search (BM25 + vector) with linear ranking
• hybrid-with-gbdt: Same retrieval + advanced GBDT ranking (slower but best quality)
• deepresearch: Exhaustive search with 10,000 retrieval targets (maximum recall)
• deepresearch-with-gbdt: Exhaustive search + GBDT ranking (slowest, most thorough)

Pro tip: The quality difference between hybrid and hybrid-with-gbdt can be dramatic for complex queries. The GBDT model offers significantly better relevance at the cost of 2-3x higher latency. For research tasks where you need to find everything relevant, try deepresearch variants which cast a much wider net!

Manage Your Data

NyRAG also gives you simple tools for cleanup. Open the advanced menu (three-dot icon ⋮ in the top right) and you will find two cleanup actions. Clear Local Cache removes cached files for all projects on your machine, which is useful when you want to re-process from scratch locally. Clear Vespa Data deletes the indexed documents in Vespa for the project, which is useful when you want a clean index before re-feeding. Both actions ask for confirmation so you do not delete data by accident.

Bonus: Try Web Crawling Mode

In addition to local documents, NyRAG supports web crawling. By switching to the web_example project, you can point NyRAG at a website and have it crawl, extract, and index content automatically.

Switch to web crawling mode: Select web_example (web) from the dropdown at the top and open the configuration editor. If you are currently on the chat screen, open the three-dot menu (⋮) and choose “Edit Config” to bring the editor back. From there, update the same credential fields as you did for doc_example, then click “Start Indexing” to crawl and feed the site.

Description: Shows web crawling in progress with terminal logs displaying discovered URLs and processed pages

Web Mode Features: Web mode discovers and follows links automatically, while still respecting robots.txt and crawl delays so you do not hammer a site. It also does smart content extraction to drop navigation and boilerplate, deduplicates very similar pages, and supports resume so you can continue a crawl after interruption.

Example Use Cases: Web mode is a good fit for product documentation, knowledge bases, blog archives, help-center content, and technical wikis. In general, it works best on sites with consistent HTML structure and clean, text-heavy pages.

Tips: Start small. Crawl a limited part of a site first so you can sanity-check what gets extracted and indexed, then expand. Use exclude patterns to skip sections you do not want (for example /pricing or /sales/*), and keep an eye on the terminal output panel so you can spot loops, unexpected URLs, or pages that fail to parse.

Troubleshooting

Running into issues? We’ve got you covered! For detailed troubleshooting guides covering Vespa connection errors, LLM configuration, document processing, and more, see the Troubleshooting section in the main README.

Quick help: If you get stuck, the fastest path is usually to ask in the Vespa Slack community, where people can help you interpret logs and query behavior. If you think you found a bug or want to request an improvement, open an issue in GitHub Issues. And when you want deeper background on schema, ranking, and deployment, the Vespa Docs are your go-to reference.

Conclusion

Congratulations! You now have a working RAG app: a Vespa Cloud deployment that can retrieve high-quality context, and a small UI that lets you ingest data and chat with it.

Building a high-quality RAG system is never trivial. There are multiple moving parts to get right: the quality of the LLM, the size and management of its context window, and how effectively your retrieval system surfaces the most relevant information.

Thanks to the out-of-the-box Vespa RAG blueprint on Vespa Cloud, much of this complexity is handled for you. It comes with multiple ranking profiles, and its default hybrid retrieval setup combines vector similarity with BM25 text matching, ensuring your LLM sees the best possible context for every query.

Vespa Cloud doesn’t just make building RAG easier, it makes it scalable, fast, and reliable, giving you production-ready infrastructure, auto-scaling and observability without the headaches of self-hosting. Whether you’re experimenting with small datasets or scaling to millions of documents, Vespa Cloud provides the tools and flexibility to make your RAG project shine.

Want to dive deeper? Start with the RAG Blueprint Tutorial for a thorough conceptual walkthrough. And remember the Vespa Slack community is always there to help. Ask questions, share what you’ve built, or get advice on retrieval, ranking, and deployment strategies.

Ready to experience the power of Vespa Cloud for yourself? Sign up today and start building high-quality RAG applications with ease!

This month, we’re announcing several updates focused on retrieval quality, ranking flexibility, and developer productivity. Each feature is designed to help engineering teams build faster, more accurate, and more maintainable retrieval and ranking systems, while giving businesses better relevance, lower operational overhead, and more predictable performance at scale.

Let’s dive into what’s new.

Product updates

• Announcing the Vespa.ai Playground
• The Vespa Kubernetes Operator
• Faster result rendering with CBOR
• Pyvespa 1.0 with improved HTTP performance
• Hybrid search relevance evaluation tool
• Configurable linguistics per field
• “switch” operator in ranking expressions
• Vespa is now available on GCP Marketplace
• Feed data and run queries in the Vespa Console

Announcing the Vespa.ai Playground

The Vespa Playground is a new GitHub space where we share projects, tools, and demos built on the Vespa platform. It’s a practical place to explore real examples for embeddings, model training, and feed connectors that you can clone, run, and build on your own.

These repos are ideal for experimentation, learning, and inspiration, though they aren’t officially supported product releases.

Explore the Playground

The Vespa Kubernetes Operator

The safest, most robust and cost effective way to run Vespa is to deploy on Vespa Cloud, but for various reasons that’s not an option for everybody. For those who want to run Vespa securely at scale but can’t use Vespa Cloud we have now released the Vespa Kubernetes Operator. This brings many of the Vespa Cloud features such as security out of the box, dynamic provisioning, autoscaling and automated upgrades to your own Kubernetes environments.

Read more in the Kubernetes Operator documentation.

Faster result rendering with CBOR

Query result sets can be large, and increasingly so when the client is an LLM retrieving many chunks for model context. Layered ranking is designed to address this by extracting the most relevant content. Still, in some cases, the total latency is dominated by the time it takes to send the query response. Compressing with gzip can help, but is also CPU-intensive and slow.. From Vespa 8.623.5, json response generation is over twice as fast as before.

Another new option in this release is to use the CBOR format for query results. CBOR is a binary format so it can be serialized faster and produces smaller payloads, especially when the result contains lots of numeric data. Read more in the Query API reference and query performance guide.

Pyvespa 1.0 with improved HTTP performance

We have released the first major version of Pyvespa! This release switches the HTTP-client used by Pyvespa, from httpx to httpr, which gives big performance gains, especially for serializing and deserializing tensors, largely by taking advantage of the new CBOR serialization support in Vespa.

On preliminary benchmarks, we compared end-to-end latency for:

1. Vespa 8.591.16 + Pyvespa v0.63.0 (using JSON)

2. Vespa 8.634.24 + Pyvespa v1.0.0 (using CBOR)

The latter was ~4.9x faster when returning 400 hits with a 768-dim vector each. Performance gains will be smaller when not returning large result sets with tensors, but still significant. You may encounter different exceptions than before, but we strived to not change any user-facing API’s even if we bumped the major version.

Go to Pyvespa

Hybrid search relevance evaluation tool

Hybrid search combines lexical and embedding based search to get the best from both. One of the tasks you need to solve is to pick an embedding model that provides a good quality vs. cost tradeoff for your use case. We have done a systematic evaluation of modern alternatives in this blog.

The code used to run these experiments is now merged into Pyvespa. You can use the VespaMTEBApp to evaluate embedding model performance on any task/benchmark compatible with the mteb-library. See example usage from the tests.

Configurable linguistics per field

Vespa now lets you specify linguistics profiles on fields to select some specific linguistics processing in your Linguistics module. In Lucene Linguistics, linguistics profiles map to analyzer configuration, optionally in combination with a specific language.

For example, you can define a Lucene analyzer like this in services.xml:

  <item key="profile=whitespaceLowercase;language=en">

    <tokenizer>

      <name>whitespace</name>

    </tokenizer>

    <tokenFilters>

      <item>

        <name>lowercase</name>

      </item>

    </tokenFilters>

  </item>

And use it in the schema, under any field’s definition, like this:

field title type string {

  indexing: summary | index

  linguistics {

      profile: whitespaceLowercase

  }

}

By default the linguistics profile will be applied both when processing the text of the field and the text searching it, but you can also specify a different linguistics profile on the query side, which is useful for e.g. doing synonym query expansion.

We’ve added a sample application demonstrating how to use multiple Lucene linguistics profiles across multiple fields and updated the Vespa linguistics documentation with usage examples.

New “switch” operator in ranking expressions

We have added a “switch” function in ranking expressions as a clearer, more maintainable alternative to deeply nested if() clauses, making complex ranking easier to read, debug, and evolve.

switch (attribute(category)) {

    case "restaurant": myRestaurantFunction(),

    case "hotel": myHotelFunction(),

    default: myDefaultFunction()

}

Learn more

Vespa is now available on GCP Marketplace

Vespa Cloud is now listed on the GCP Marketplace, making it easier to deploy and manage Vespa using native Google Cloud billing and procurement. Vespa Cloud is already available on AWS Marketplace.

See details

Feed data and run queries in the Vespa Console

The onboarding experience is now even smoother for new Vespa Cloud users. When you follow the getting started guide and deploy a sample app from the browser, you can immediately feed data and run queries directly in the browser. This makes it easy to try your own data and see how it behaves in Vespa.

We also provide examples showing how to do the same using pyvespa, the Vespa CLI, or curl.

Try it Free

New content and learning resources

We published several new articles and resources since our last newsletter to help teams get more out of Vespa and stay ahead of new developments in search, RAG, and large-scale AI.

Examples and notebooks:

• playground.vespa.ai

Videos, webinars, and podcasts

• How To Scale AI in Digital Commerce Effectively
• 2025 Year in Review

Blogs and ebooks

• Clarm: Agentic AI-powered Sales for Developers with Vespa Cloud
• Embedding Tradeoffs, Quantified
• Enterprise AI Search vs. the Real Needs of Customer-Facing Apps
• Eliminating the Precision–Latency Trade-Off in Large-Scale RAG
• How Tensors Are Changing Search in Life Sciences
• The Search API Reset: Incumbents Retreat, Innovators Step Up
• Why AI Search Platforms Are Gaining Attention
• Why Life Sciences AI Is a Search Problem (Part 5 of 5)
• Why Life Sciences AI Is a Search Problem (Part 4 of 5)

Upcoming Events

Lightning Lesson: Personalized Relevance with VLMs and Sparse Vectors: February 17, 11:30am ET

• Intro to sparse vectors and tensors for efficient data handling
• Using Vision-Language Models (VLMs) to extract high quality and nuanced features from images
• Leveraging these features in sparse representations for hyper-personalized search & recommendations

Register Now

February 18: The Zero Results Problem in eCommerce

• 🔗 10am CET (EMEA): Save your spot
• 🔗 1pm ET (Americas): Save your spot

March 11: The Relevance Problem in eCommerce

• 🔗 10am CET (EMEA): Save your spot
• 🔗 1pm ET (Americas): Save your spot

March 10: Vespa Q1 Product Update

• 🔗 10am CET (EMEA): Save your spot
• 🔗 1pm ET (Americas): Save your spot

👉 Follow us on LinkedIn to stay in the loop on upcoming events, blog posts, and announcements.

Thanks for joining us in exploring the frontiers of AI with Vespa. Ready to take your projects to the next level? Deploy your application for free on Vespa Cloud today.

AI is transforming quickly. What started with Q&A chatbots has already evolved into deep research applications and, now, autonomous AI agents. Vespa is proud to be at the center of this shift, enabling some of the most proficient adopters of AI, such as Perplexity. To help organizations maximize the benefits of Vespa, we’re building a robust partner ecosystem. These partners help bring Vespa’s AI-native capabilities into real-world deployments across industries.

Meet the innovators shaping the future of AI. Today’s spotlight: Nexla

Nexla + Vespa.ai: The Power Duo for AI-Ready Data Pipelines

When AI systems fall short, it’s rarely the model’s fault. It’s the messy reality of data spread across systems and never quite staying in sync. That’s why Nexla and Vespa partnered together.

Nexla makes data usable.

Vespa makes data intelligent at scale.

Together, they turn messy, distributed enterprise data into real-time AI search, recommendation, and RAG systems, without months of custom code gluing things together.

Nexla: Making Enterprise Data Usable

Nexla is an enterprise-grade, AI-powered data integration platform that turns raw data from any source into production-ready data products. It provides a declarative, no-code way to move, transform, and validate data across ETL/ELT, reverse ETL, streaming, APIs, and RAG pipelines.

Think of Nexla as the layer that answers: “How do we reliably get the right data, in the right shape, to the systems that need it?

Core capabilities:

• 500+ Bidirectional Connectors: Pull data from databases, APIs, cloud storage, SaaS apps, and data warehouses, including systems like Salesforce, Snowflake, and Amazon S3.

• Metadata Intelligence: Nexla automatically scans sources and generates Nexsets, virtual, ready-to-use data products with schemas, samples, and validation rules. Example: If a price field suddenly switches from numeric to string, Nexla detects it before bad data reaches production search.

• Express (conversational pipelines): A conversational AI interface where you can simply describe what you need. Example: You can say, “Pull customer data from Salesforce and merge with Google Analytics,” and it builds the pipeline for you.

• Universal integration styles: Supports ELT, ETL, CDC, R-ETL, streaming, API integration, and FTP in a single platform.

Nexla processes over 1 trillion records monthly for companies like DoorDash, LinkedIn, Carrier, and LiveRamp.

Vespa: Where Retrieval Becomes Reasoning

Vespa is a production-grade AI search platform that combines a distributed text search, vector search, structured filtering, and machine-learned ranking in a single system.

Think of Vespa as the engine that answers: “Given all this data, how do we retrieve, rank, and reason over it in real time?”

It powers demanding applications like Perplexity and supports search, recommendations, personalization, and RAG at massive scale.

Core capabilities:

• Unified AI Search and Retrieval: Vespa natively combines vector and tensor search for semantic retrieval, full-text search for precise keyword matching, and structured filtering on attributes like categories, prices, and dates to enable richer, contextual search without stitching multiple systems together.

• Real-time Retrieval and Inference at Scale: Rather than separating indexing, ranking, and inference across multiple systems, Vespa performs real-time machine-learned ranking and model inference where the data lives. This means you can serve fresh, personalized results with predictable sub-100 ms latency even for large datasets.

• Multi-Phase Ranking and Custom Logic: Vespa lets you embed custom ranking logic, including ML models like XGBoost, directly into your search pipeline using ONNX. You can combine relevance signals, business rules, and semantic vectors in multi-stage ranking to fine-tune which results surface first.

• Massive Scalability with High Throughput: Designed for real-world, high-traffic applications, Vespa can scale horizontally across clusters, handling billions of documents with sub-100ms query latency and up to 100k writes per second per node.

• Multi-Vector and Multi-Modal Retrieval: Vespa natively handles multiple vectors per document, with support for token-level embeddings, ColPali-based visual document retrieval, and tensor-based computations for precise, cross-modal relevance and ranking.

GigaOm recognized Vespa as a leader in vector databases for two consecutive years, noting its performance advantages over alternatives like Elasticsearch, up to 12.9X higher throughput per CPU core for vector searches.

How Nexla and Vespa Work Together

The Nexla-Vespa partnership removes one of the hardest parts of AI systems: getting clean, well-modeled data into a high-performance retrieval engine, continuously.

Nexla recently launched a Vespa connector that makes data integration with Vespa seamless. The integration includes:

Vespa Connector in Nexla: Handles all data piping from sources like Amazon S3, PostgreSQL, Pinecone, Snowflake, and others directly into Vespa:

Vespa Nexla Plugin CLI: Automatically generates draft Vespa application packages (including schema files) directly from a Nexset, eliminating manual configuration:

This means you can move data from S3 to Vespa, migrate from Pinecone to Vespa, or sync PostgreSQL to Vespa, all without writing a single line of code.

When Nexla Clients Should Use Vespa

You’re a Nexla client. Use Vespa when you need:

Advanced AI search and RAG applications: If you’re building intelligent search, recommendation systems, or RAG applications that require hybrid search (combining semantic vector search with keyword matching and metadata filtering), Vespa is purpose-built for this. Nexla gets your data into Vespa, while Vespa delivers production-grade AI search with machine-learned ranking.

Real-time, high-scale query performance: When you need to serve thousands of queries per second across billions of documents with sub-100ms latency, Vespa’s distributed architecture scales horizontally without compromising quality. Nexla ensures your data flows continuously into Vespa with incremental updates and CDC support.

Complex ranking and inference: If your use case requires multi-phase ranking, custom ML models, or LLM integration at query time, Vespa executes these operations locally where data lives, avoiding costly data movement. Nexla prepares and transforms your data into the exact schema Vespa needs.

Cost efficiency at scale: Vespa delivers 5X infrastructure cost savings compared to alternatives like Elasticsearch while handling vector, lexical, and hybrid queries. Nexla minimizes integration costs by automating pipeline creation and schema management.

When Vespa Clients Should Use Nexla

You’re a Vespa client. Use Nexla when you need:

Multi-source data consolidation: Vespa is your search and inference engine, but data lives everywhere, S3 buckets, PostgreSQL databases, Snowflake warehouses, Salesforce CRMs, APIs, and files. Nexla connects to 500+ sources with bidirectional connectors and consolidates data into Vespa without custom ETL scripts.

Automated schema generation and management: Instead of manually writing Vespa schema files and managing schema evolution, Nexla’s Plugin CLI auto-generates schemas from your Nexsets. As source schemas change, Nexla’s metadata intelligence detects changes and propagates them downstream automatically.

Data transformation and enrichment: Before data hits Vespa, it often needs cleaning, filtering, enrichment, or format conversion. Nexla provides a no-code transformation library and supports custom SQL, Python, or JavaScript, all without maintaining separate ETL infrastructure.

Vector database migration: Moving from Pinecone, Weaviate, or another vector database to Vespa? Nexla handles the migration with zero code, extracting records, transforming data to match Vespa’s schema, and syncing documents continuously.

Data quality and monitoring: Nexla continuously monitors data flows with built-in validation rules, error handling, and automated alerts. When data quality issues arise, Nexla quarantines bad records and provides audit trails, ensuring Vespa always receives clean, trustworthy data.

Real-time and streaming pipelines: Vespa supports real-time updates, but getting real-time data from streaming sources (Kafka, APIs, databases with CDC) requires integration logic. Nexla handles streaming, batch, and hybrid integration styles, optimizing throughput and latency for each source type.

Conclusion

Nexla solves data readiness.

Vespa solves intelligence and precision at scale.

Together, they give teams a clean, practical path from raw enterprise data to real-time AI applications. Vespa gives you production-grade vector search, hybrid retrieval, and RAG capabilities at any scale. Nexla eliminates months of pipeline development and makes multi-source data flows conversational.

Ready to explore?

Start at express.dev for conversational pipeline building, or explore the Vespa connector in Nexla’s platform to see how quickly your data can power real AI applications.

Overview

Clarm helps open source software companies convert GitHub stars into revenue through AI-powered lead generation, content production, and developer support automation. When building their platform, Clarm needed a search engine that could power accurate, zero-hallucination AI responses while handling complex enrichment across millions of GitHub data points. They chose Vespa for its unified text, vector, and structured search capabilities and were able to deploy to production in under a day.

The Problem: Software / OSS Companies Struggle to Monetize

“Most OSS founders can’t get attention for their software initially. They’re so focused on building the product that marketing, SEO, and content creation get dropped. We built Clarm to automate all the growth work founders drop so they can focus on git commits,” explains Marcus Storm-Mollard, founder and CEO of Clarm.

The challenge is fundamental: 99% of successful open source is funded by businesses paying for solutions, but early-stage OSS companies lack the infrastructure to identify, engage, and convert those potential paying customers. They have thousands of GitHub stars but no clear path to revenue.

Clarm addresses this through three product pillars:

1. Lead Generation & Prospecting: The killer feature. Clarm takes repo data from customers and competitors, enriches it with signals from website visits, commits, issues, and community engagement, then ranks and identifies good-fit prospects and potential enterprise buyers.

2. Marketing & Content Production: Automated content creation from commits, PRs, and codebase analysis, helping OSS companies maintain consistent technical marketing.

3. Developer Support Automation: AI-powered support across Discord, Slack, GitHub Issues, and websites, with deep integrations and analytics for scaling customer success.

The Search Challenge

At the core of all three pillars sits a critical technical requirement: accurate, explainable search and retrieval.

“We realized early that search, not generation, was the real problem to solve. Generating LLM answers isn’t hard. Finding the right information to base them on is everything,” Marcus notes.

Clarm needed a search engine that could:

• Handle hybrid retrieval (combining text search, vector embeddings, and structured filters)
• Power zero-hallucination AI responses grounded in verifiable context
• Process and rank millions of GitHub data points in real-time
• Support complex multi-signal enrichment for lead scoring
• Scale cost-effectively on a startup budget

Traditional vector databases like Supabase or search engines like Elasticsearch couldn’t deliver the unified, production-grade retrieval required for Clarm’s zero-hallucination architecture.

The Solution: Vespa’s Production-Grade Hybrid Search

Marcus discovered Vespa after researching how companies like Perplexity and Onyx built their advanced retrieval systems.

“We really liked that Vespa started as a search engine and evolved into a vector-based system. It made so much sense for what we were building. Vespa’s ranking and tensoring are built in, so we know our results are accurate and relevant right out of the box,” Marcus explains.

Rapid Deployment: Less Than One Day to Production

Clarm began experimenting with Vespa’s Docker image for local development, then transitioned to Vespa Cloud for production deployment during their Y Combinator batch.

“It took about half a day to set up how we wanted it. That speed of onboarding made a huge impact during YC. We just deployed it, and it worked,” Marcus recalls.

The quick deployment was critical. Clarm was racing toward Demo Day and couldn’t afford weeks of infrastructure setup. Vespa’s unified approach eliminated the complexity of stitching together multiple systems for text, vector, and structured search.

Key Vespa Capabilities Powering Clarm

• Unified Retrieval Pipeline
  • Single query endpoint combining text search, vector similarity, and structured filters - no need to orchestrate multiple databases or services.
• Built-in Ranking & Tensor Operations
  • Native support for complex ranking models and tensor operations means Clarm can implement sophisticated lead scoring without custom ranking layers.
• Real-Time Indexing
  • GitHub events, user interactions, and enrichment signals are instantly searchable, enabling live lead intelligence and up-to-date AI responses.
• Scalable Cloud Deployment
  • Automatic scaling and high availability handled by Vespa Cloud, allowing Clarm’s two-person engineering team to focus on product features instead of infrastructure operations.
• Developer-Friendly Architecture
  • Docker-based local development, straightforward schema design, and comprehensive documentation enabled rapid prototyping and iteration.

The Results

Clarm’s decision to build on Vespa Cloud delivered immediate impact:

• <1 Day to Production: From prototype to live search infrastructure deployed during YC
• Zero-Hallucination Architecture: Accurate retrieval enabling trustworthy AI responses grounded in verifiable data
• High-Quality Lead Intelligence: Sophisticated ranking of GitHub data points across 50K+ collective stars from customers like Better Auth (23.3K stars) and Cua (11.3K stars)
• Exceptional Support: Direct collaboration with Vespa’s engineering team throughout development

“The setup was easy, the support from the Vespa team was incredible, and everything just worked. We didn’t need to look anywhere else,” Marcus emphasizes.

Customer Success: GitHub Stars Becoming Revenue

Clarm’s customers are seeing measurable results from the AI-powered lead generation platform:

• Better Auth: Grew from 8K to 23.3K GitHub stars in 3 months with Clarm’s lead gen and engagement automation
• c/ua: Scaled from 5K to 11.3K stars while identifying and converting enterprise prospects
• Skyvern AI: after struggling with after hitting 19k stars, reduced support workload by 94% with Clarm across Github, Discord, and Slack
• Engagement Depth: Developers “pair programming” with Clarm’s AI agents for extended sessions, sending thousands of queries a day and sessions lasting up to 22 hours

What’s Next: Building the Future of OSS Monetization

Clarm represents a new category of growth infrastructure built specifically for software and open source companies. By combining Vespa’s production-grade retrieval with their own zero-hallucination agent framework, Clarm is proving that AI-powered sales and marketing can be trustworthy, explainable, and grounded in truth.

“We’re focused on proving product value and retaining customers right now. Everything depends on us growing our customers’ MRR and showing software and OSS companies they can build sustainable businesses,” Marcus shares.

That focus is reflected in Clarm’s positioning: “You build awesome software. Now build a business.” It resonates with software founders who want to monetize without compromising their community values. By recognizing that a vast majority of successful open source is ultimately funded by businesses paying for solutions, Clarm offers a clear path forward: free software for the community, paid solutions for enterprises.

Conclusion

Clarm’s architecture reinforces a lesson many teams learn the hard way: LLMs are only as reliable as the retrieval systems behind them. By treating retrieval as a first-class system, built on Vespa Cloud, Clarm unified text search, vector similarity, structured filtering, and ranking into a single production-grade platform, eliminating the fragility and guesswork common in vector-only stacks.

The result is an agentic AI platform that can reason over live data, explain its outputs, and scale predictably without stitching together multiple databases or post-hoc ranking layers. This foundation enabled a small team to move from prototype to production in days, operate across millions of GitHub signals, and help open source companies turn community adoption into sustainable revenue.

More importantly, Clarm’s success offers a blueprint for any organization building serious AI applications: when retrieval is reliable, ranking is expressive, and data is always fresh, AI systems become trustworthy enough to power real business outcomes. Clarm is building the future of OSS monetization, and Vespa is the retrieval engine making it possible.

The typical approach: open the MTEB leaderboard, find the “Retrieval” column, sort descending, pick something that fits your size budget. Done, right?

Not quite. MTEB doesn’t tell you:

• How fast is inference on your actual hardware?
• What happens when you quantize the model weights?
• How much quality do you lose with binary vectors?
• Does this model even work well in a hybrid setup?

So we ran the experiments ourselves. We picked models from the MTEB Retrieval leaderboard with these criteria:

• Under 500M parameters (practical for most deployments)
• Open license
• ONNX weights available (required for Vespa)
• At least 10k downloads in the last month (actually used in production)

For each model, we benchmarked across:

• Model quantizations (FP32, FP16, INT8)
• Vector precisions (float, bfloat16, binary)
• Matryoshka dimensions (for models that support it)
• Real hardware (Graviton3, Graviton4, T4 GPU)
• Hybrid retrieval (semantic, RRF, and score normalization methods)

Spoiler: We found some really attractive tradeoffs - 32x memory reduction, 4x faster inference, with nearly identical quality.

What MTEB doesn’t show you

Model quantization

Vespa uses ONNX runtime for embedding inference. Most models on HuggingFace ship with multiple ONNX variants - here’s Alibaba-NLP/gte-modernbert-base as an example:

Lower precision weights = smaller model = faster inference. But how much faster, and what’s the quality hit?

• On CPU: INT8 models run 2.7-3.4x faster while keeping 94-98% of the quality
• On GPU: INT8 is actually 4-5x slower than FP32. Don’t do this.

The difference between 30ms and 100ms query latency is huge. If you’re on CPU, INT8 is often a no-brainer.

On GPU, use FP16 instead - you get ~2x speedup with no meaningful quality loss.

GPU vs CPU: The T4 GPU runs 4-7x faster than Graviton3 for embedding inference. If you’re processing high query volumes or doing bulk indexing, GPU may be worth it.

Vector precision

Model quantization affects inference speed. Vector precision affects storage and search speed. Different knobs, both important.

Here’s the math for 100 million 768-dimensional embeddings:

That’s a 32x difference between FP32 and binary. When memory is what forces you to add more nodes, this matters a lot.

bfloat16 is free: In our benchmarks, bfloat16 vectors show zero quality loss compared to FP32 - it’s a 2x storage reduction you can take without any tradeoff.

Matryoshka dimensions

Some models support Matryoshka Representation Learning (MRL) - you can truncate the embedding to fewer dimensions and still get decent results. Fewer dimensions = less storage, faster search.

Here’s EmbeddingGemma at different dimension sizes:

Source: EmbeddingGemma paper

Interestingly, EmbeddingGemma actually scores higher at 512 dimensions than at 768. We didn’t dig into why - it may be an artifact of the smaller evaluation set - but it’s a reminder that more dimensions isn’t always better.

Not all models support this - check the model card before truncating. If it wasn’t trained for MRL, slicing dimensions will tank your quality.

Inference speed

If you have a 200ms latency budget and your embedding model takes 150ms, you’re in trouble. We benchmarked actual inference times so you can plan accordingly.

We measured two things for each model:

1. Query latency - how long to embed an 8-word query
2. Document throughput - embeddings per second for 103-word docs

Tested on three AWS instance types:

• c7g.2xlarge - Graviton 3 (ARM CPU)
• g4dn.xlarge - T4 GPU
• m8g.xlarge - Graviton 4 (ARM CPU)

These numbers are pure ONNX inference time. Your actual indexing throughput will also depend on HNSW config and existing index size, but embedding inference is usually the bottleneck.

Quality

We evaluated all models on NanoBEIR, a smaller but representative subset of the BEIR benchmark. This let us run a lot of experiments without waiting forever.

For each model, we measured nDCG@10 across four retrieval strategies:

• Semantic only - pure vector similarity
• RRF (Reciprocal Rank Fusion) - combines BM25 and vector rankings
• Atan hybrid - normalizes scores using arctangent before combining
• Linear hybrid - linear normalization before combining

The hybrid methods consistently outperform pure semantic search. Every single model in our benchmark scored higher with hybrid retrieval than semantic-only. On average, the best hybrid method beats semantic-only by 3-5 percentage points. That’s a meaningful lift you get “for free” by just using BM25 alongside your vectors.

We also tested each model with binarized vectors (int8). This is where things get interesting:

• ModernBERT models barely flinch - Alibaba GTE ModernBERT retains 98% of quality (0.670 binary vs 0.685 float)
• E5 models take a bigger hit - E5-base-v2 drops to 92% (0.602 binary vs 0.651 float), and E5-small-v2 to just 87%

The takeaway: not all models are created equal for binary quantization. The newer ModernBERT-based models handle it much better than the E5 family. Make sure to check before assuming you can just binarize everything.

Interactive leaderboard

We built an interactive leaderboard so you can explore the full results yourself. Filter by hardware, sort by different metrics, and expand each model to see the full breakdown across dimensions and precisions. Open in full screen.

Getting started with Vespa

Ready to put this into practice? Here’s how to configure an embedding model in Vespa:

<component id="alibaba_gte_modernbert_int8" type="hugging-face-embedder">
    <transformer-model model-id="alibaba-gte-modernbert"/>
    <max-tokens>8192</max-tokens>
    <pooling-strategy>cls</pooling-strategy>
</component>

Here’s a schema with a binarized embedding field (96 dimensions = 768 bits packed):

schema doc {
  document doc {
    field id type string {
      indexing: summary | attribute
    }
    field text type string {
      indexing: index | summary
      index: enable-bm25
    }
  }
  field embedding_alibaba_gte_modernbert_int8_96_int8 type tensor<int8>(x[96]) {
    indexing: input text | embed alibaba_gte_modernbert_int8 | pack_bits | index | attribute
    attribute {
      distance-metric: hamming
    }
    index {
      hnsw {
        max-links-per-node: 16
        neighbors-to-explore-at-insert: 200
      }
    }
  }
}

And a rank profile using linear normalization for hybrid scoring:

rank-profile hybrid_linear {
  inputs {
    query(q) tensor<int8>(x[96])
  }
  function similarity() {
    expression {
      1 - (distance(field, embedding_alibaba_gte_modernbert_int8_96_int8) / 768)
    }
  }
  first-phase {
    expression: similarity
  }
  global-phase {
    expression: normalize_linear(bm25(text)) + normalize_linear(similarity)
    rerank-count: 1000
  }
  match-features {
    similarity
    bm25(text)
  }
}

Check out the embedding documentation for full details on configuration, including how to set up binary quantization and hybrid search.

Going further

Binary vectors are fast - really fast. Vespa can do ~1 billion hamming distance calculations per second, roughly 7x more than prenormalized angular distance. That speed difference means you can crank up targetHits significantly and still stay within latency budget. More candidates evaluated = better recall. So binary vectors aren’t just about 32x storage savings - they give you headroom to tune for quality too.

And luckily, Vespa’s phased ranking architecture lets you make up for any remaining quality loss in later phases. You can retrieve candidates with hamming distance, then rescore in any of the following ways:

• float-binary - Use float for query vector, and unpack the bits of document vector to float for angular distance calculation. Example
• float-float - Retrieve with hamming distance but rerank with full-precision vectors paged in from disk. Should be limited to a small candidate set.
• int8-int8 - Same as float-float, with int8 vectors (scalar quantization, not to be confused with binary quantization) for both query and document. Faster and more storage-efficient than float-float, with a small precision cost.

See this great huggingface blog post for more details on these techniques.

For even better results, add a cross-encoder reranker as a final stage. Or (especially if you have several user signals or features), train a GBDT model to learn optimal combinations.

The beauty of Vespa’s ranking expressions is that you can mix and match all of these - BM25, a bunch of other built-in features, vectors, rerankers, learned models - however you want.

A few caveats

Multilingual support

If you need to support multiple languages, your options narrow. The multilingual-e5-base model handles 100+ languages but comes with a quality tradeoff compared to English-only models. For English-only workloads, stick with the specialized models.

Context length

Document length matters too. Many newer models handle 8192 tokens, EmbeddingGemma can take 2048, while the E5 family tops out at 512. If your documents are long, look at benchmarks like LoCo (Long Document Retrieval) - NanoBEIR won’t tell you much here.

For long documents, check out Vespa’s layered ranking - it lets you rank chunks within documents so you’re not forced to return irrelevant chunks from top-ranking docs.

Test on your own data

NanoBEIR is a good starting point, but your domain matters. A model that tops the leaderboard on scientific papers might struggle with product descriptions, legal documents, or your internal knowledge base.

Benchmark rankings can be misleading for specialized domains. The models we tested were trained on general web data - if your corpus looks very different (medical records, source code, niche industry jargon), the relative rankings might shuffle significantly.

We’ve open-sourced the benchmarking code in pyvespa so you can run the same experiments on any model with any dataset compatible with the MTEB library. Swap in your own data and see how different models actually perform for your use case.

Consider finetuning

If off-the-shelf models underperform on your domain, finetuning can help significantly. Even a small set of query-document pairs from your actual data can boost relevance.

Tools like sentence-transformers make this straightforward. The ROI is often worth it for production systems where a few percentage points of nDCG translate to real user impact.

Wrapping up

The “best” embedding model depends entirely on your constraints. But now you have real data to make that call:

• Cost sensitive? Binary quantization with a compatible model (like GTE ModernBERT) gives you 32x savings with minimal quality loss.
• Running on CPU? INT8 model quantization speeds up inference 2.7-3.4x.
• Need great quality? Alibaba GTE ModernBERT + hybrid search is hard to beat.
• Latency-critical? E5-small-v2 with INT8 can do a query inference in only 2.5ms on Graviton3.

The interactive leaderboard above has all the details. Explore, filter, and find the sweet spot for your use case.

For those interested in learning more about Vespa, join the Vespa community on Slack to exchange ideas, seek assistance from the community, or stay in the loop on the latest Vespa developments.

Originally posted 25th Aug. 2025 on The New Stack

In my years working across life sciences, one question comes up again and again: What’s next for AI in our field? The truth is that the life sciences industry faces challenges unlike any other.

Where a bank or retailer might deploy AI chatbots to improve customer service, our world is defined by enormous, messy datasets, including clinical trials, lab results, publications and patient records. This must be interpreted with care. The stakes are not just efficiency or convenience; they are breakthroughs in treatment, safety and patient outcomes.

That’s why I believe the real opportunity for generative AI (GenAI) in the life sciences is not in chatbots, but in enabling deep and precise retrieval. Success here means connecting across multiple sources, reconciling heterogeneous data and surfacing insights that a human researcher would struggle to piece together.

Imagine asking: “Find me colorectal cancer trials using ZALTRAP [a drug] with the most recent supporting publications.” GenAI, when applied effectively, can handle that complexity, and this is where the next frontier begins.

From Traditional Search to AI-Driven Discovery

For decades, search in life sciences has mostly meant keyword lookups or rule-based retrieval. Researchers, clinicians and pharma teams relied on these tools to sift through scientific literature, clinical trial data, patents and regulatory filings. They worked well enough for simple, well-defined questions. But as soon as you needed to account for domain-specific language, synonyms or the complex relationships between diseases, molecules and pathways, traditional search hit its limits.

The result? Endless manual refinements, stitching insights together from different sources and lots of time spent just finding the right information.

Now, with GenAI and large language models (LLMs), that’s changing. LLM-powered search understands meaning, not just exact words. You can ask complex, natural-language questions and get results that connect the dots across literature, trials and patents — even when they use different terminology. This opens up entirely new ways of working: identifying drug repurposing opportunities hidden in disconnected studies, accelerating biomarker discovery or finding previously unseen links between biological entities. It’s faster, more comprehensive and far less manual.

Why Tensors Matter in This Shift

Life sciences data comes in all shapes and sizes — omics data, 3D protein structures, medical images, regulatory documents, clinical trial reports and more. Most of it is unstructured or semi-structured, which makes it tricky for AI systems to find and assemble relevant information quickly. Given the nature of life sciences, accuracy is critical. “Good enough” seldom exists.

This is where tensors come in.

So, what is a tensor? Think of it as a multidimensional data container. A vector is a one-dimensional list of numbers. A matrix is two-dimensional. A tensor goes beyond that, capturing multiple dimensions at once. This allows AI models to represent complex relationships — like spatial configurations of proteins or contextual relationships between words in a scientific article — even if those pieces of information are far apart.

In other words, tensors let AI “see” and learn patterns that are deeply embedded across different dimensions of data.

Tensors in Action: Protein Structures

Take structural biology as an example. Models like AlphaFold use 3D tensors to represent the spatial relationships between amino acids. These tensors allow the model to learn how proteins fold, twist and interact — crucial knowledge for understanding disease mechanisms and designing new therapies.

When you embed a protein as a tensor, you preserve:

• Sequential data (the order of amino acids)
• Spatial relationships (how parts of the protein fold and connect)
• Biochemical properties (like charge or hydrophobicity)

This rich representation lets machine learning (ML) models predict protein folding, identify binding sites, map protein-protein interactions and even discover new drug targets.

The same idea applies beyond proteins.

Medical imaging, for example, can use tensors to encode not just pixels, but also their contextual relevance, helping AI detect subtle cancer markers even in noisy scans. In clinical settings, tensors help AI analyze data streams from wearables or Internet of Things (IoT) devices in real time, enabling faster interventions.

Beyond Retrieval: AI Agents in Life Sciences

AI agents are another emerging application. Think of them as intelligent assistants that continuously gather, analyze and synthesize information across fragmented data sources. An AI agent could monitor new literature, clinical trials and regulatory updates in real time, summarize findings and even suggest next research steps.

Good agents don’t just fetch information — they connect it, building context and reason through problems step by step.

The key here is multistep reasoning. Good agents don’t just fetch information — they connect it, building context and reason through problems step by step.

This means faster reasoning, better accuracy and more meaningful insights. This allows you to stitch together multimodal data and ask questions across modalities and time. For example, as illustrated in the example below, you can now find patients for trial recruitment for a disease subtype based on certain progression (or regression) images over time. You can do this by combining into a tensor the patient’s medical record, biomarker assays, histopathology slides and any other prognosis outcome notes.

Why This Matters

Life sciences are moving into an era where data is simply too complex and too vast for traditional tools. Tensors provide the foundation for AI models to handle this complexity, enabling everything from better search to advanced reasoning. Whether it’s predicting protein structures, extracting insights from clinical data or powering AI agents that help researchers focus on discovery rather than data wrangling, tensors are quietly becoming the backbone of the next wave of AI in life sciences.

Originally posted 7th Nov. 2025 on The New Stack

The search landscape is shifting. In recent months, Microsoft announced the retirement of the Bing Search API, while Google limited its own API to a maximum of 10 results per query. These moves mark a notable change in the way the web’s dominant search providers view access to their data and who gets to build on it.

For more than a decade, search APIs like Bing and Google Custom Search have been part of the web’s plumbing. Developers have used them to retrieve web results, images and news without maintaining their own indexes. Enterprises have embedded them in applications such as customer support, knowledge bases and market intelligence to provide external context. Startups and research teams have used them to collect training data, ground language models or perform competitive analysis without running their own crawlers.

In short, search APIs have offered a simple way to access the open web programmatically, bridging the gap between consumer search and enterprise information retrieval.

The AI Shift

The emergence of generative AI has changed what the search infrastructure needs to deliver. With retrieval-augmented generation (RAG) becoming central to AI systems, developers now require flexible retrieval layers within the AI pipeline, not just returning links.

Against this backdrop, the timing of Microsoft and Google’s decisions stands out. Microsoft has folded search access into Azure’s AI stack through its Grounding with Bing Search feature for AI agents, while Google continues to reduce external visibility into its own results. Limiting queries to 10 results per call fits with its long-standing goal of minimizing bulk data extraction and automated scraping.

The business thinking is clear: Both companies are steering developers away from large-scale, open retrieval and toward AI-mediated access inside their own ecosystems. Full result sets are expensive to serve and often used by automated systems such as SEO platforms, data-mining tools or research crawlers rather than by interactive users. Restricting APIs helps contain those costs while repositioning web data as a controlled resource for higher-level AI services.

A Reset, Not a Retreat

This isn’t a collapse of search, but a realignment of control. The open, list-based APIs of the past belong to an era where raw results were the product. In the generative AI era, incumbents are redefining search around answers, grounding and context, tightly coupled with their cloud ecosystems.

But as the large providers step back, new players are moving in. Perplexity and Parallel represent a new generation of search APIs designed for AI workloads. They publish benchmarks, expose APIs openly and emphasize retrieval quality and low latency, the performance characteristics that matter most in RAG and agentic systems. You can read more about the Perplexity search API here.

Perplexity has also shown that it outperforms Google on relevance for RAG-style tasks. Not to be outdone, Parallel, founded by Twitter’s former CEO, Parag Agrawal, recently reported better results than Perplexity, using Perplexity’s own evaluation tool.

A Hot Market, New Foundations

The search API market is heating up again, this time around AI native infrastructure. Beneath Perplexity and Parallel is a common component: Vespa, the open source engine built for large-scale retrieval, ranking and machine learning inference.

Vespa’s role in these systems reflects a broader shift in architecture: Search infrastructure is now part of the AI stack itself. As models depend more on retrieval, factors such as performance, scalability and the ability to combine structured and unstructured data have become key differentiators.

The incumbents are narrowing access; the innovators are expanding it. Either way, search is once again at the center of how the web is organized, only this time, it’s being rebuilt for AI.

Originally posted 13th Oct. 2025 on The New Stack

Gartner’s new “Market Guide for Enterprise AI Search” offers a valuable perspective on how generative AI is transforming enterprise knowledge access. Yet, as the focus remains on internal productivity and governance, a crucial question arises: Do the same approaches apply when search is customer-facing and directly tied to engagement, conversion and revenue?

Gartner’s Take on Enterprise AI Search

Gartner’s research focuses on internal, employee-focused use cases, including digital workplace assistants, IT support, HR knowledge management and compliance automation. The goal is to enable AI assistants and copilots to increase employee productivity by synthesizing information across corporate data silos. Customer experience (CX) scenarios are mentioned only briefly and treated as adjacent markets rather than core priorities.

As a result, the report emphasizes governance, connectors, metadata enrichment, security trimming and knowledge graphs. These capabilities are critical for enterprise environments but less relevant to real-time, customer-facing systems.

Performance is primarily assessed in terms of reliability and governance for AI assistants, which aligns with the internal focus of enterprise AI search. In these settings, latency and throughput matter less than access control and compliance. Hybrid search and retrieval-augmented generation (RAG) are recognized as core capabilities, but the emphasis remains on managing complexity across diverse data silos rather than serving high-volume, low-latency workloads.

Gartner also expects large enterprises to operate multiple embedded search platforms across SaaS suites such as Microsoft 365, Salesforce and SAP. This architecture fits internal knowledge management well, but is not designed for customer-facing systems, where performance, scale and accuracy directly shape user experience and business outcomes.

Customer-Facing Search

This creates a clear gap for organizations building customer-facing AI applications, where search performance is not just a productivity factor but a business-critical capability. Whether e-commerce, finance, media, social networks, market intelligence or other apps, the requirements are fundamentally different. Developers must deliver high-volume, low-latency retrieval and ranking, often processing thousands of queries per second under strict service-level objectives.

These systems depend on multiphase ranking pipelines, sophisticated tensor computation and multimodal retrieval across text, image and structured data. They must also serve generative or retrieval-augmented results at machine speed. Indexing and feature updates occur in near real time to reflect changes in inventory, behavior or content streams, all while maintaining uptime, cost efficiency and performance at scale.

Unlike internal enterprise AI search systems, where governance and policy compliance are the top priorities, customer-facing AI search must optimize for low-latency relevance, responsiveness and personalization.

These systems are embedded directly into the core product experience, influencing not just user satisfaction but also revenue, engagement and retention. They require a unified architecture that can handle both lexical and vector retrieval, execute learned ranking models close to the data to reduce network bandwidth and support large-scale inference workloads without heavy orchestration or additional middleware.

Conclusion

Gartner’s report provides an essential framework for understanding enterprise AI search within the context of internal productivity and governance. However, the same assumptions and architectures do not translate to customer-facing applications.

As the market evolves, a more precise distinction will emerge between enterprise AI search platforms designed for internal knowledge synthesis and AI search platforms built for production-grade, real-time environments where search is the product itself.

The latter must meet far higher expectations for performance, scale, accuracy and multimodality, which are defining characteristics of modern generative and retrieval-augmented systems.

For organizations building customer-facing AI applications, Vespa provides an alternative to the traditional “insight engine” lineage. It is built for AI-native systems, not retrofitted enterprise search. Vespa powers high-volume, real-time retrieval and ranking at scale for companies such as Perplexity.ai, Spotify and Yahoo. Its architecture supports multiphase ranking, tensor computation, multimodal retrieval and serving AI at machine speed. Vespa is designed for engineering-led companies where search is the product and performance directly drives revenue.

Originally posted 3rd Oct. 2025 on The New Stack

Retrieval-Augmented Generation (RAG) systems constantly face a trade-off: Precise results often mean higher latency and cost, while faster responses risk losing context and accuracy. The solution isn’t choosing one or the other. It’s redesigning retrieval. Let’s explore three techniques that together eliminate this trade-off: multiphase ranking, layered retrieval and semantic chunking.

When combined, they create a retrieval stack that balances speed, scalability and precision.

Multiphase Ranking: Incremental Refinement of Results

At the heart of retrieval lies ranking. Running deep machine learning (ML) ranking across the entire candidate set introduces increased latency and increases infrastructure cost. On the other hand, lightweight scoring methods alone can’t capture enough context, so precision suffers. That’s why a multiphased approach provides a balanced alternative.

Instead of choosing between expensive deep models or fast but shallow heuristics, multiphase ranking stages scoring from cheap to costly. Lightweight filters (lexical, approximate nearest neighbor or ANN) quickly trim the candidate pool, while progressively heavier ML functions are applied only to the top results. This preserves precision while keeping latency and compute under control.

Multiphase ranking provides a balanced alternative:

• Phase 1: Fast filtering using keyword matching or ANN search.
• Phase 2: Reranking with dense embeddings, hybrid similarity measures or custom ranking expressions.
• Phase 3: Advanced machine-learned models, personalization signals or domain-specific scoring rules.

This staged refinement ensures that expensive models are applied only where they add the most value.

Benefits include:

• Cost-aware precision: Spend compute strategically across phases.
• Hybrid logic: Blend symbolic rules, semantic similarity and behavioral data.
• Personalization: Adapt results to individual users or sessions.

By mirroring these best practices in large-scale search and recommendation, multiphase ranking enables RAG systems to deliver accurate results without breaking latency budgets.

Layered Retrieval: The Foundation of Ranking Quality

Even the most sophisticated multiphase ranking stack can’t compensate for poor retrieval units or noisy inputs. The quality of ranking depends heavily on the retrieval unit you choose:

• Fine-grained chunks (paragraphs or sliding windows) maximize recall, since even short queries are likely to match. But they introduce trade-offs:
  • Context fragmentation: Key signals get split across chunks.
  • Redundancy: Overlapping chunks inflate index size and cause duplicates.
  • Downstream burden: Ranking and large language models (LLMs) must stitch fragmented evidence together, increasing token usage and latency.
• Whole-document retrieval preserves global context and reduces redundancy, but often sacrifices precision. Large spans of irrelevant text are pulled into prompts, diluting relevance signals, inflating token costs and making reranking less effective.

A well-designed retrieval strategy typically lands in between: defining a semantic retrieval unit that captures enough local context to be self-contained, while still preserving structural metadata (headings, sections, timestamps) that downstream ranking can exploit. This balance ensures that ranking operates over high-quality candidates, minimizing wasted compute and maximizing the signal-to-noise ratio that feeds the LLM.

Layered retrieval achieves this balance by combining both levels of relevance:

1. Rank and select the most relevant documents.
2. Within those documents, retrieve only the top-K chunks.

This hierarchical process preserves the broader context of document-level signals while narrowing down to the specific spans that matter.

Benefits include:

• Reduced token usage and lower prompt costs.
• Cleaner, more coherent context for the LLM.
• Improved precision without sacrificing recall.

Semantic Chunking: Precision Starts With Preprocessing

Finally, retrieval quality depends on how you index your data. Long-form documents stored as monoliths often produce noisy retrieval, because only part of the content is relevant to a given query.

Semantic chunking addresses this by splitting documents into meaningful, self-contained units like paragraphs or logical sections while retaining contextual metadata like headings, authorship or timestamps.

Benefits include:

• Higher recall: More granular entry points into documents.
• Better precision: Irrelevant sections can be excluded at query time.
• Metadata enrichment: Supports symbolic filtering and downstream ranking.

Chunking can increase index size and requires careful prompt assembly, but when combined with layered retrieval and multiphase ranking, it becomes a powerful foundation for precision.

Building a Production-Ready Retrieval Stack for RAG

Together, these three techniques address the biggest pain points in scaling RAG:

• Overlong prompts from including too much content.
• Context fragmentation from isolated chunks.
• Rigid ranking pipelines that ignore domain logic.

A robust retrieval stack should therefore:

• Index documents with semantic chunking while preserving metadata.
• Retrieve hierarchically through layered retrieval.
• Refine results efficiently with multiphase ranking.

This combination enables more accurate, cost-efficient and trustworthy LLM outputs, especially when paired with retrieval-aware prompt engineering.

Final Thoughts

As RAG systems scale, retrieval design becomes a key differentiator. Moving beyond simple vector or ANN search to incorporate multiphase ranking, layered retrieval and semantic chunking dramatically improves both efficiency and output quality.

Vespa was built to handle these retrieval challenges at enterprise scale. Its tensor-native architecture supports multiphase ranking, layered retrieval and semantic chunking directly in-cluster, eliminating external bottlenecks and costly workarounds. By running retrieval and ranking where the data lives, Vespa delivers low-latency, high-precision results across billions of documents and thousands of queries per second.

Whether you’re building knowledge assistants, research agents or large-scale production RAG systems, Vespa provides the retrieval foundation that keeps generative AI accurate, efficient and ready to scale.

Originally posted 29th Aug. 2025 on The New Stack

A few years ago, my daughter told me that her school research project was so deep she had to venture all the way to Page 3 of Google. That moment stuck with me because it shows just how ingrained search has become in our lives. “To google” quickly turned into a verb. Search was built for human speed: deliver a shortlist of results, let the user scan, interpret, decide and even go to Page 3 if required. This same foundation has powered e-commerce, content discovery, compliance and countless other applications.

Generative AI has changed expectations almost overnight. Instead of typing keywords, people ask questions in plain language and increasingly complex ones. They expect search not just to return accurate results, but to do the heavy lifting: answer a question, summarize research or even solve a problem.

Generative AI Is Maturing Fast

Despite being a relatively recent phenomenon, there are already at least three levels of GenAI maturity:

• Level 1: Chatbots – “Answer my question.”
• Level 2: Deep research – “Research this and report back.”
• Level 3: Agentic systems – “Solve my problem.”

At Levels 2 and 3, retrieval becomes challenging. Systems may run dozens of searches for a single task. A sluggish retrieval layer doesn’t just slow things down; it can cripple the whole experience.

The Challenge: Delivering Retrieval Accuracy at Scale

Vector databases made similarity search possible, enabling large language models (LLMs) to ground answers in large unstructured data sets. But vector search alone isn’t enough. Production-grade AI search needs more: combining semantic, keyword and metadata retrieval, applying machine-learned ranking and handling constantly changing structured and unstructured data, all at scale.

Trying to bolt these components together across multiple systems quickly hits its limits. Bandwidth, integration overhead and shallow connections create bottlenecks and erode accuracy, which is key since people rarely question the answers the AI provides.

Enter the AI Search Platform

The AI search platform is a new class of infrastructure that makes retrieval smarter, faster and more scalable by uniting classical search techniques with modern AI: vector and tensor search in embedding spaces, full-text search for precision, multistep ranking and real-time inference, using machine-learned models and tensor math. It enables accurate search at machine speed with filtering and ranking to ensure only the most relevant answers surface instantly. The AI search platform is critical in simplifying the development and deployment of generative AI at every maturity level.

Why This Matters for Enterprises

Mainstream data platforms, such as Snowflake or Postgres, now include basic vector search capabilities. That’s fine for entry-level GenAI chatbots, but not for customer-facing deep research or agentic AI use cases where speed, scale and accuracy deliver competitiveness.

For CIOs, this has created a split:

• Basic enterprise GenAI: supported by incumbent platforms, “good enough” for simple internal tasks.
• Advanced enterprise GenAI: for demanding customer-facing use cases, where only AI search platforms can keep up.

In this landscape, pure-play vector DBs risk being marginalized, sandwiched between incumbent data platforms for simple use cases, and AI search platforms that deliver scale, performance and accuracy.

Companies that adopt AI search platforms early will set the pace in this new era. Search is no longer just a utility; it’s becoming the backbone of AI-driven business. And no doubt, the backbone of my daughter’s research.