Agentic systems usually have two jobs: build context, then use that context to produce an answer or action.

Many failures that look like LLM problems start in the context-building step. The answer the LLM gives is limited by the context it was given or it finds through tool calls. If the agent model cannot find the right sources then improving the generation model will not improve the overall system.

Many failures that look like LLM problems start in the context-building step.

A client, Specstory, wanted to give users the ability to ask questions from agent history. For example, why a team chose Authlib for authentication and what alternatives they considered. The chatbot needs the right prior conversations, decisions, and tradeoffs from a large corpus of coding sessions. The model and system prompt help only after those chat turns are retrieved and in context.

The same pattern showed up in an AnkiHub operator review in our private community. A request for help studying based on lecture slides only works if the agent’s tool calls retrieve the right flashcards.

The setup changes by product. A human may see retrieval as a search box. An agent may call local search, semantic search, a database, or a SaaS API. Either way, the process is the same: gather the right context, then generate from it.

An order-status agent might pull customer records from Salesforce, shipment data from SAP, and tracking events from a logistics API. Those sources are structured application state rather than a document corpus. The agent still has to gather the right context before it answers.

A coding agent runs rg, opens files, reads logs, and inspects tests before writing a patch. A research agent searches the web and internal notes before writing an answer. A study assistant searches deck facts and user context before suggesting what to learn next.

When context building fails the symptoms look like generation failures.

A useful source can disappear at several points in the path. The trace should tell you which one.

A better model helps with reasoning and writing, but it cannot give a better answer without the right context.

The Mixedbread OfficeQA-Pro Eval shows the pattern at benchmark scale. OfficeQA-Pro uses 89,000 pages of financial documents, dense tables, scanned PDFs, and questions that require reasoning across documents. Giving Codex better search tools reduced tool calls and improved answer quality.

Plain text tools like grep and rg work (ish) over flat code files. They do not work well when context lives in PDFs, tables, chat histories, multi-modal inputs, web results, SaaS records, and permissioned data. In those cases the agent needs retrieval that can combine exact terms, meaning, metadata, permissions, and ranking quality.

Retrieval needs traces and evals

Once retrieval enters the architecture, the next question is whether it finds the right information.

For that you need traces and evals. For each retrieval step the minimum trace is the input, the outputs, and a way to label whether each output was relevant.

For a coding agent using rg the input is the command, the output is the returned snippets, and the label says which snippets helped, which were noise, and which relevant files were missing.

For product retrieval the step might be BM25, semantic search, hybrid search with reranking, a generated SQL query, or something else. Capture the query or arguments, the returned documents or chunks, and whether those results were helpful.

Trace each retrieval step by itself, then evaluate the full context-building pass. The local trace answers “Did this query return useful material?” The full trace answers “Did the system collect everything the model needed before generation?” If it did, failures are a generation problem. If not, it’s a retrieval problem.

You cannot know where the failure started or what to fix without traces.

Different failures need different fixes

“Improve retrieval” is too broad to be useful because different problems have different solutions. If a relevant document is missing the trace should show where it disappeared: query building, retrieval, filtering, ranking, or final context assembly.

That diagnosis should land on a concrete system change. The right fix depends on what the system was trying to retrieve. A decision-history question needs the decision, the alternatives, and the chats where the team worked through them. A study question depends on the lecture material, deck metadata, semantic matches, and the user’s study context.

The architecture

Once you trace individual retrieval calls, the full architecture has a simple shape: fan out to context-building tools, then fan in to generate the final output.

Where Vespa fits

Vespa is useful once retrieval has to serve many filtered, ranked requests instead of one broad search call. The app still owns the user request, workflow, and final context budget. Vespa stores the retrievable chunks and serves filtered, ranked evidence back to the app.

That boundary matters because agents rarely make one clean search. They ask follow-up questions, widen or narrow filters, and need enough trace data to explain why a source did or did not reach the model.

Consider a research agent answering, “What changed in the refund policy after the February outage?” The answer might live across the current policy doc, a support ticket thread, an incident review, and a changelog. The app knows the user, tenant, allowed sources, time window, and why the agent is asking. Vespa serves the evidence.

The same shape applies when the retrievable units are customer records, shipment events, product rows, or permissioned SaaS objects instead of document chunks.

In Vespa, that starts with storage. For policy docs, notes, tickets, chat histories, and long PDFs, the retrievable unit is usually a chunk with a stable source id, chunk id, timestamp, permission fields, metadata, and embedding. For records and SaaS objects, it might be a row or object with the same retrieval fields. The schema makes those fields available for search, filters, ranking, and summaries.

At query time, the app sends the request text, filters, query embedding, and rank profile. Permission and tenant filters narrow the candidate set before ranking. Vespa can retrieve candidates with text matching, nearest-neighbor search, or both. Ranking can favor the current policy doc, exact issue IDs, and semantically related incident notes.

Document summaries return the chunk text, ids, metadata, scores, and source fields the app needs. The app chooses what enters the model context. For debugging, the trace should preserve the query, filters, hits, selected context, answer id, and reviewer labels. Vespa provides query traces and logs; the labeling workflow is usually app-owned.

Ranking

Ranking is where Vespa decides which allowed candidates deserve the limited context budget.

Better ranking improves precision (more of the returned chunks are relevant). If the right chunk is ranked 40th and the context only includes the top 10, the system behaves as if retrieval missed it. Returning the top 40 is one answer but it costs a lot of tokens and latency. If the relevant chunks are near the top the system can pass fewer chunks into the model, spend fewer tokens, reduce latency, and give the model less noise.

Semantic similarity is one ranking signal. A chunk with a strong vector score may still be less useful than a newer source, an authoritative source type, or a chunk with an exact error code, file name, ticket ID, or quoted phrase. Permissions and tenant boundaries should already have narrowed the candidate set. Ranking decides how to order the allowed candidates.

A ranker usually uses several kinds of information:

The Vespa schema defines the document fields and ranking options. A rank profile is a named scoring recipe in that schema. A profile can combine exact text scores, semantic scores, metadata, values sent with the query, and model scores in one formula.

Vespa separates the query that finds candidates from the rank profile that orders them. A nearestNeighbor query uses the vector index to find candidate chunks near the query embedding. A text query uses term matching. A hybrid query can use both. After that, Vespa scores each candidate so it can return the best ones first. For example, a hybrid rank profile might start with vector closeness plus BM25, then use a slower model such as a cross-encoder to rescore the best candidates.

Scale introduces more challenges

With a small corpus a prototype can search and still feel fast. With millions or billions of chunks, every extra retrieval call, candidate, ranking pass, and returned token adds up.

One user request can trigger several retrieval calls in an agentic system. Each of those could be multi-stage retrieval calls (query rewriting, keyword/semantic, ranking, etc). Combine that with many concurrent users and it becomes clear the retrieval layer has to keep quality high while controlling latency and cost.

When retrieval and ranking run in the same serving path, filters can shrink the candidate set before expensive ranking, summaries can limit returned text, and traces can show whether latency came from matching, ranking, or response size.

Multi-stage retrieval is the common shape

Most production systems split retrieval into stages, even when the UI looks like one chat box. The trace should show whether the app asked the wrong retrieval question, the candidate set never included the source, filters removed it, ranking buried it, the summary left out needed fields, or context assembly dropped it before generation.

If the query builder chose the wrong filters then changing the embedding model will not help. If the right source was retrieved but ranked low, the ranking must be tweaked. You must be able to drill into any part of the system. That’s much easier if it’s all in one system

Vespa references

Whatever stack you use, when an answer is wrong you should be able to see whether the source was never retrieved, filtered out, ranked too low, or dropped before generation. Vespa is strongest when that diagnostic path needs to run over large, changing corpora with ranking, filtering, and serving constraints built into the same retrieval layer.

In this post, we want to tell you about our summer internship at Vespa, and what we managed to build. We will quickly explain what the Model Context Protocol is, and what its benefits and limitations are, before showing off our implementation’s capabilities and discussing our experiences.

What is MCP?

So, what actually is the Model Context Protocol? Plainly, it is an open standard for how to connect AI assistants to external data sources, systems, and environments. Prior to MCP, letting an agent interact with a system meant a custom integration for that specific agent to that specific system. Now, you design and implement the service once, and then any compatible LLM or AI agent can connect to the system.

MCP enables this by standardizing the interface between a system and any assistant that wants to use it. A service (Vespa Cloud in our case) implements an MCP server and exposes capabilities in three distinct ways:

• Resources: read-only context for the LLM to understand the system
• Tools: executable actions the assistant can invoke to actually operate the system
• Prompts: reusable templates or shortcuts for structuring interactions with language models

Our implementation mainly focuses on implementing tools and letting an LLM take advantage of the right resources. Later, you can read about why designing tools is both difficult to get right and important for how an assistant uses an MCP server.

This is not the first time people at Vespa have looked into the possibilities of the Model Context Protocol. Last year’s interns implemented a server that ships with the application package, and there also exists an internal tool that runs locally, which served as great inspiration. What makes our approach unique is that the server is standalone and can be publicly hosted, meaning that any user with a Vespa Cloud account can connect to the server and have access to their own tenants and applications.

The distinction matters more than it might first appear. An application-specific server has to be shipped together with one application and typically only sees that application. This means that every application that wants an MCP connection has to ship a new server. A locally run MCP server requires installation and needs to be kept up to date on each user’s machine. Our server requires neither. It runs as a publicly hosted service, requiring no install step and no separate server to run each distinct application. The tradeoff, discussed later, is that a publicly hosted server is currently unable to securely conduct most Data Plane operations, and this is why our server currently targets the Control Plane.

MCP Tool design

Designing good MCP tools requires many considerations. Working with a non-deterministic model makes it much harder to predict what even minor changes to tools will do to the end result. Anthropic provides a variety of guidelines for designing tools, which we have used, along with our own experiences, to shape our toolset. We will walk through the design of the vespa_get_logs tool as an example of our design process.

Before an LLM can use a tool, it must know it exists and how to call it. In MCP, this is provided by the Tool Description. Our experience, reflecting that of others, was that small wording and format changes could significantly improve the likelihood of correct tool use. A good example is the time_range parameter in our vespa_get_logs tool.

Our first implementation required an agent to supply epoch milliseconds directly, giving the timeframe in absolute terms. This mostly worked, but the agent was prone to miscalculations. Using relative expressions like “now-1h”, and falling back to ISO 8601 datetime strings when needed, seemed to help. Most queries are relative anyway, and such strings are likely more frequent in agents’ training data than raw epoch integers. We learned that you should prefer parameter formats that match the use case and the model’s training data.

Protecting the agent’s context window is also critical. A naive implementation of the log tool would return the entire log for a given timeframe straight from the Vespa Cloud API. Logs produce large volumes of data, most of which is irrelevant to any given question. Feeding all of it to the agent bloats the context window, which slows down reasoning and drives up cost, without necessarily improving the answer. To address this, the tool lets the LLM choose how much pre-processing it wants: a detail parameter toggles between a compact summary and raw log lines, and a severity filter excludes lower-priority entries like info logs. This way, the agent can start with a lightweight summary and only request the full detail if the situation calls for it.

This culminated in the following tool description:

@mcp.tool(...)
async def vespa_get_logs(...):
    """Search a deployed instance's logs. Defaults to warning-and-above from the last hour.

    Repeated events (identical except for their timestamp) are collapsed into
    one entry carrying `count` and `first_seen`. The response is capped to a
    fixed size budget; when it overflows, the oldest events drop first.

    Args:
        start_time:
            Start of the time range.
            Accepts relative expressions ("now-1h", "now-7d") or ISO datetime strings.
            Defaults to "now-1h".
        end_time:
            End of the time range.
            Accepts the same formats as start_time. Defaults to "now".
        min_level:
            Minimum severity, most-to-least severe:
            fatal, error, warning, info, config, event, debug, spam.
            Pass "info" for normal logs, None for all.
        contains:
            Case-insensitive substring over service, component, and message.
        limit:
            Max events returned, most recent kept (default: 100, range: 1-100).
            Values outside this range are clamped and reported in the
            response warnings.
        detail:
            "concise" (time, level, service, message)
            or "detailed" (adds host, pid, component).
        ...
    """

Both point to the same conclusion: good tool design is less about exposing everything an API can do, and more about shaping the interface so the agent makes good decisions when deciding how to use tools.

Benefits

Now, why would having a publicly hosted Vespa MCP server be useful? One might think that just handing an LLM the Vespa CLI and a terminal would give it the same capabilities as having an MCP server. That is not quite true. What separates the two approaches is control, and as it turns out, capability as well.

Let’s start with control. A tool-based interface lets us define how an agent is able to use the access we provide. We can decide what actions are possible, instruct on when to use certain tools, and ensure that unwanted actions are impossible. This control starts before a tool is even called. Each tool defines the exact structure of the input it expects, making it easier for the agent to know how the tool should be used in the first place. This results in fewer attempts needed to call it correctly, as opposed to a trial-and-error approach towards the CLI.

Structuring available tools and giving the necessary context also guides the LLM toward how to solve problems in the first place. As we will see later, this also results in the agent making fewer mistakes. It will always know what actions are available to it and have information about how to use different tools. On top of this, the server chooses how to present error messages, meaning that tools fail informatively and hand the assistant the next step for fixing the problem, rather than the model having to interpret raw error messages.

An MCP server is not limited to wrapping existing commands; it can expose entirely new capabilities built specifically for how an assistant uses it. Our tools surface Grafana metrics directly, giving the assistant real observability data with no current CLI equivalent. More on what possibilities this provides is elaborated on later in the post.

Lastly, a server is convenient. Connecting an LLM to a hosted MCP server is basically plug and play. Meaning that both new and experienced developers can start using it in a matter of minutes. For new Vespa devs, having an MCP server to help your AI assistant poke at Vespa, explain features, and teach you how to design applications becomes invaluable. For the experienced dev, letting the LLM handle time consuming jobs, such as debugging why an application has poor quality of service, opens up time for other tasks that are more important.

MCP Limitations

During our development of the MCP server, we experienced some protocol limitations we had to work around.

The first case occurred when designing the tool for deploying applications. An application package consists of several files that define the application; node structure, schemas, authentication, etc. The challenge was how the LLM should upload these files to the MCP server. There exists no built-in way to upload files in MCP today, so we ended up exploring several possible solutions.

Our first attempt let the LLM upload a JSON object with a dictionary mapping from file names to file contents. This approach was simple and required only one tool call, making it fast for a single deployment. On the other hand, small tweaks to applications required the LLM to rewrite the entire package, and uploading binary files, such as embedding models, was not possible. Therefore, we had to look into other solutions.

Most agents have access to some sort of terminal, either locally or in a sandbox environment. Therefore, we assume that the agent is able to zip the application package, but how do we send this over the MCP server? We tried Base64 encoding the files and using them as input for the tool. Copying encoded data is, however, something LLMs struggle to do accurately and quickly. So, our second attempt wasn’t satisfactory either.

Our solution was to avoid the protocol altogether. Instead, the tool returns a unique URL to a separate HTTP endpoint for uploading the package. Then the LLM can zip the package and use curl to upload. On one hand, this means more tool calls and slower execution for a single deployment. On the other, it ends up being faster with several deployments, as it does not have to rewrite the entire package to make small adjustments. From our experience, several deployments are often needed, making us prefer this solution over the others.

The second limitation we faced while developing the server was also related to the deployment tool. The time it takes to deploy a Vespa application can vary from seconds to several minutes. Therefore, we had to decide if the MCP tool should return immediately and have the LLM poll for the deployment result with a separate tool, or block until the deployment is finished.

Letting an LLM poll the status means many tool calls per deployment, costing valuable tokens, and often resulting in the assistant just setting a timer after a few attempts. The deployment might, however, finish early during the timer, leaving the LLM to do unnecessary waiting. In contrast, letting the tool block might keep the agent from working in parallel. It also gives no feedback to the user, leaving room for confusion as to why the tool appears frozen.

We decided that having the tool block was best, but added a timeout as a parameter so it doesn’t wait too long. This way, there is some feedback along the way, and the LLM can choose to do other tasks while deploying. It still requires several tool calls for a deployment, so the solution is not optimal. During the summer, a new mechanism for “long running tasks” was added to MCP, which seems to solve our problem. Once this is adopted by most MCP clients, the deployment tool will probably be refactored to use this feature.

Using the MCP server

Using the MCP server, we could now create a Vespa application from scratch only through prompts. In this case, we started with a simple prompt about wanting a blog application with Vespa serving as the search engine. In a few minutes the model created the Vespa Cloud application and a web page, then connected the two. To tune the search, we just needed to specify what results we wanted to surface for a given query, and the model would make appropriate changes to the schema and redeploy.

This showcases one of the more interesting use cases for our MCP server. Simply explain to the LLM what kind of application you want, and in a matter of minutes it is up and running. Perfect for new Vespa developers who want to get started, understand how Vespa works, and see what possibilities it provides.

Extending the capabilities with metrics

Thus far, most of the functionality of the MCP server can also be done using the Vespa CLI. The next step was to add something beyond what Vespa CLI can do currently. In the Vespa Cloud Console, there is a Grafana metrics dashboard for each application, with hundreds of metrics for any time window. Each metric panel has a short explanation of how to interpret the data. Giving the MCP server access to all of these metrics is useful for debugging and monitoring applications.

To implement this, we first looked at what API endpoints Vespa has for metrics. We quickly realized that all the endpoints only gave the metrics at the current time, with no way of looking at the metrics for the last week or month. The capability of looking at metrics for a specified time would require storing all metrics, for all timestamps, for all applications, at all times. This approach was unnecessarily complex, and it would essentially mean duplicating the existing functionality of Grafana.

Instead, we started looking at how the Vespa Cloud dashboard gets all its metrics. After some reverse engineering, we found the endpoints used to get the structure of the dashboard and also how to query the actual metrics data. Then, the only thing needed was to add some tools so that the LLM could get access to the panels with descriptions and data, directly from Grafana.

When this was implemented, we wanted to see what it could help us do. We started by consulting an LLM on whether or not our application needed to be scaled up. After looking at several metrics of resource usage, it correctly decided that it does not need upscaling.

Next, we tried to restart one of the content nodes of an application, to see if the LLM could explain the sudden increase in the number of active documents. After 17 tool calls checking different metrics, deployments, and logs, the LLM was able to correctly identify that a node was restarted, in addition to explaining that it was likely done by a Vespa Cloud host.

Looking through metrics data manually can be time-consuming, but from our testing, the LLM seems to be good at crunching through the numbers. This shifts the role of the developer from diagnosing to fixing. We are excited to see if this can be useful in real world scenarios.

Authentication Issues and Server Hosting

An internet-exposed MCP server requires tight security, as anyone can send requests to it. We have to treat each incoming request as potentially malicious, while also serving legitimate requests. Vespa Cloud handles authentication in two different ways, based on the type of information you want to access. Our server needed both in order to get the functionality we wanted.

The Control Plane was the easy half. It authenticates with Auth0, so we could re-use the same logic used by the Vespa Cloud Console, where one authentication flow gives the user access to their applications. The MCP Server would act as an OAuth proxy in front of Auth0, where clients register with the MCP server before they’re handed off to Auth0 for the actual authentication.

The Data Plane is where things got messy. To query a Vespa Cloud application, you need a separate token, scoped per application, so the Auth0 magic simply doesn’t work here. Our workaround, for testing purposes, was to hardcode a pre-generated token into the server. It worked for the single tenant we tested against, but wouldn’t scale to multiple users from different tenants.

The solution for this was not quite obvious, so we explored some options. The simplest fix would be to require the tools to provide a token parameter; however, this is blatantly insecure and the MCP spec disallows it. Another was to mint temporary Data Plane tokens using your Control Plane authentication; however, this would require changes to how tokens are minted, which was out of scope for our summer project. We ultimately settled on our workaround until a cleaner solution is available.

Authentication to third party APIs (e.g., the Vespa Cloud API) seems to be an ongoing discussion in the MCP protocol, with recent developments like URL elicitation representing early attempts at addressing it.

One of the goals we had for this project was to host the MCP server ourselves. During the summer, the 2026-07-28 MCP spec was released, which removed protocol level sessions along with the initial handshake. This change made hosting considerably easier. With this change, any request could be served by any instance, so the server deploys and scales like more traditional HTTP APIs, instead of needing the sticky routing or shared state that was required previously.

Does it actually help?

Handing an LLM a big toolbox does not automatically mean it will use it well, or even better than it would have without the toolbox at all. We therefore wanted to find out:

Does having this MCP server actually help an LLM operate a Vespa Cloud application better than just giving it the Vespa CLI and a terminal?

Evaluating that, however, turned out to be easier said than done. There’s rarely one correct sequence of tool calls for a task. For example, an LLM might discover a schema by first listing applications or simply by jumping to vespa_list_schemas if it has already guessed the application’s name, and both can be perfectly good. Worse, an LLM’s behaviour would change from run to run. Two different runs on the same suite, with the same code, might yield two completely different results, making the “correct solution” even harder to define. Any single run is a weak signal, and the “answer” is not something you can hard-code.

So, we grade two ways. Deterministic assertions check the outcome against the real Vespa Cloud API: does the application exist? Are its nodes healthy? Does the deployed schema contain the fields it should? An LLM judge scores the process against a per-scenario rubric: did the agent reach for the right concepts? Did it fix the actual bug rather than paper over it? We found that neither is sufficient alone. Assertions can’t distinguish a lucky guess from sound reasoning, and an agentic judge should not be trusted with facts that can be checked deterministically.

Across three full sweeps of the evaluation scenarios, both agents mostly got there in the end. The MCP agent passed 97% of the deterministic assertions versus the CLI agent’s 95%. Seeing this, you might be tempted to think that the tools barely help.

We don’t think so, and the giveaway is the number of tries it took to get there. Agents using the MCP server averaged less than half the number of attempts before deploying successfully. Further, the use of MCP resulted in more scenarios passing cleanly, resolving faster, and averaging a better rubric score when judged by another LLM.

Tools don’t make the model smarter; they make it wrong less often.

A determined enough model will eventually find the correct path to make it work, regardless of interface. However, the MCP server buys you fewer wrong turns, even if it costs a few more tokens.

A model typically needs to discover the tools before using them, resulting in actions that don’t produce task progress. The extra cost comes from every tool’s description having to be held in context on every turn, whether or not they are used. An assistant using the CLI carries none of this overhead. It already knows or guesses the commands and has no need to discover them or keeping track of the descriptions. However, mistakes are not free either: a malformed deploy still costs time and tokens, and has to be fixed afterwards.

Meaning, it is not so much “cheap CLI” versus “expensive MCP” as it is two ways of spending the same budget. Either up front on discovery and structure, or on the back end recovering from mistakes. We’d expect that trade to favor the MCP server more as tasks get harder and wrong turns become more costly.

What’s missing

The most obvious missing part is proper Data Plane access. The workaround we created limited us to one hardcoded tenant, so it’s not ready for production. That said, our explorations suggest a Control Plane only server is feasible without too much additional effort.

Furthermore, the whole server is undergoing a security review to ensure it is up to our standard before a possible public release. We are also taking a harder look at what tool guardrails we should include to ensure safe usage.

Lastly, we need feedback from real people using the MCP server. One of the hardest challenges in creating the server has been figuring out what tools to include, and feedback from actual Vespa Cloud users would help a lot.

Our Summer at Vespa

After a summer spent exploring the Model Context Protocol, with both its limitations and strengths, we end with a working MCP prototype that lets you deploy, discover, inspect, and observe your Vespa Cloud applications. We have had to solve problems facing a continuously evolving protocol and implement functionality not already existing in the current ecosystem. Overall, we are proud of what we have created, and hope that it can be of use to Vespa Cloud developers in the near future.

Finally, we would like to thank all our colleagues at Vespa for trusting four interns with a surprising amount of freedom and production access, as well as a lot of patience for all of our questions. We have had the opportunity to learn an incredible amount this summer, and we wish the whole team further good luck.

This month, we’re shipping updates that give you time-constrained ANN search, sub-query ranking support, flexible provisioning, new rank features and telemetry export.

Let’s dive into what’s new.

Calling All Vespians: Vespa.ai Live Has Landed

As more teams build with Vespa, bringing the community closer together has become a major focus for us. Earlier this year, we ran our first virtual meetups, extending our Slack community and were joined by more than 100 Vespians from around the world - from the US and Ukraine to Singapore, Australia, Kazakhstan, Egypt, and beyond. But while virtual events are great, nothing quite compares to meeting in person - learning from peers, exchanging ideas, and continuing the conversation over coffee, beer, or wine. That’s why we’re excited to announce our first in-person community meetup: Vespa.ai Live!

Vespa.ai Live brings together the engineers, architects, researchers, and practitioners building modern retrieval and ranking systems. The event includes technical sessions, real-world user stories, expert panels, interactive unconference discussions, and plenty of opportunities to connect with others building in this space. Leading authors Trey Grainger and Doug Turnbull will also be in the lineup, sharing their perspectives on where the industry is heading. Optional pre-event training the day before includes beginner and advanced tracks to help attendees sharpen their Vespa skills before the main event.

Most of all, Vespa.ai Live is intended to be community-driven - where Vespians share lessons learned and boldly go beyond the frontier of modern search.

Learn more about Vespa Live!

Product updates

Meet latency targets more easily with time-bounded vector search

The standard HNSW algorithm for vector search explores the graph to find the closest vector matches, using exploration parameters like targetHits and filterFirstExploration to determine how extensively it searches. You often need to tune these parameters to achieve a particular latency budget, which can be challenging.

What’s new: Vespa now provides a simpler alternative where you set the latency budget and let the algorithm find the best matches within that time and then return.

To use it, set the new ranking.matching.anntimebudget query parameter to the amount of time you want to allow for ANN search. Vespa will stop searching when it reaches that limit, eliminating the need to tune exploration parameters to indirectly meet a latency target.

Pro-tip: Use the new query_approximate_nns_time metric to track the time spent on ANN. Because this metric only counts queries that actually perform ANN searches, its count dimension gives you the precise number of queries that performed ANN searches! This way, you can monitor how many queries actually perform ANN searches and how many fall back to an exact search.

Keep clusters scaling when cloud providers’ capacity is limited with max-cost-factor

In Vespa Cloud, resource specifications configure settings like vCPU, memory and disk for a cluster. Example:

<resources vcpu="32" memory="64Gb" disk="1900Gb">

Cloud providers may not always have the instance types needed to match your resource specifications. When the required resources are unavailable, Vespa isn’t able to add the necessary capacity, which blocks deployment and autoscaling.

What’s new: Vespa Cloud now lets you prioritize availability over cost when your preferred instance type isn’t available. With the new max-cost-factor setting, you can configure Vespa to select larger instance types that meet your resource requirements instead of blocking cluster re-scaling.

E.g., with a max-cost-factor=2, an instance like

<resources vcpu="64" memory="128Gb" disk="3800Gb">

matches the specification in the first example.

Note that auto migration to optimal instance types is enabled by default in Vespa Cloud for all applications, and runs in the maintenance window only, for off-peak configuration. This continuously migrates workloads to higher-performance nodes for a better cost/performance ratio, and automates the chore of optimizing your resource pool. So even if you have temporarily provisioned an oversized instance due to low availability, Vespa Cloud will move you back to the most cost efficient instance type when available.

Gain more control over how query signals affect ranking with labels

Not all parts of a query should contribute equally to ranking. Labels give you a way to identify those parts and score them independently. For example, you might want to add a smaller rank contribution for alternative forms. Labeling the parts lets you do this, and use the values in BM25 scoring, or in a general ranking expression using the raw scores.

What’s new: Vespa gives you more control over how each part of a query affects ranking. You can score BM25 for specific labeled terms, get separate BM25 scores per query-item label, or assign scores to labeled subqueries and use those scores in a ranking expression.

Scoring labeled parts of the query - bm25(field: name, label: label)

bm25(content) sums the contribution of every query term searching content. When a query is built from several parts, it is often useful to score those parts separately. This is done by attaching a label to the query items, using the YQL label annotation, and referring to that label from the rank profile:

select * from example where
  ({label:"must"}content contains "vespa") and
  ({label:"nice"}(content contains "ranking" or content contains "relevance"))

A label set on an operator enclosing others, as on the or above, is inherited by every term inside it, so a label can be given per clause instead of per term. The bm25(field: fieldname, label: label) rank feature scores only the terms carrying the given label:

rank-profile labeled {
  first-phase {
    expression: bm25(field: content, label: must) + 0.5 * bm25(field: content, label: nice)
  }
}

Since Vespa 8.738. Read more.

Scoring labeled parts of the query - bm25_for_labels(name)

Return a tensor<float>(label{}) holding one BM25 score per query item label in the indexed string field name. Each cell is labeled with the query item label and holds the sum of the BM25 scores of that label’s terms in name. This makes it possible to rank on the labeled parts of the query individually without adding one bm25(field, label) feature per label. A label gets a cell only if it actually scores, that is when at least one term carrying it searches name and matches the document.

Since Vespa 8.738. Read more.

The labeled() query operator - itemRawScore

Using labeled(subQuery, “mylabel”, 4.25) wraps an arbitrary sub-query with a label “mylabel” and score. If the sub-query matches a document, the score (4.25 in this example) can be picked up by ranking expressions using the itemRawScore rank feature. Typically used together with the rank operator:

where rank(title contains "running shoes",
  labeled(memberDiscountPercent >= 20 AND inStock > 0, "memberDiscount", 2.0),
  labeled(premiumBrand = true OR todaysFeaturedBrand = true, "premiumOrFeaturedBrand", 1.0))

A ranking profile example using the scores:

expression: bm25(title) + itemRawScore(memberDiscount) + 31.5 * itemRawScore(premiumOrFeaturedBrand)

This is similar to Lucene’s ConstantScoreQuery, often used in Elasticsearch/OpenSearch through function_score, or in Solr with ^= query syntax.

Since Vespa 8.748.3. Read more.

Build and debug custom ranking with more flexibility through new rank features

Building sophisticated ranking functions often requires working with data that isn’t readily available in the form you need, whether that means connecting values across parallel arrays or accessing the collection statistics used by text ranking algorithms like BM25.

What’s New: Vespa now has three new rank features:

• tensorFromLabelsWithOffset: constructs a tensor from an array attribute while preserving array positions. That tensor can then be used in ranking calculations.
• averageFieldLength: gives you the average length of an indexed field, which BM25 uses to normalize scores, so you can use it for debugging or custom text ranking formulas.
• queryTermDocumentFrequency: exposes the number of documents that contain each query term, which BM25 uses to calculate IDF, so you can use it for debugging or in custom ranking formulas.

These features give you more flexibility to work with structured data, understand how ranking scores are calculated, and build custom ranking logic.

tensorFromLabelsWithOffset

Creates a tensor<float> with one mapped dimension with labels from the given array attribute, and another mapped dimension where labels are constructed from the array index (offset) of that label. The cell values of the tensor are always 1.0 (suitable for multiplication). The attribute values must be integers or strings. The attribute is specified as the full feature name, attribute(name). The label-dimension parameter is required, using the same as the attribute name is common. The offset-dimension will get label value “0”, “1” and so on. This feature may be useful if there are several arrays populated in parallel, and the array index of a label is needed in order to correlate with a different array. Example: Given an attribute field myField containing the array value:

[v1, v2, v3]

tensorFromLabelsWithOffset(attribute(myField), dim, off) produces:

tensor<float>(dim{},off{}):{ {dim:v1,off:0}:1.0, {dim:v2,off:1}:1.0, {dim:v3,off:2}:1.0 }

averageFieldLength

The average length, in number of terms, of the indexed field name, as computed from the local content node index (in memory or on disk). This is the same index statistic that BM25 uses by default for field length normalization (the avg_field_len term), exposed here so it can be used for debugging and in custom text ranking formulas such as Bayesian BM25.

queryTermDocumentFrequency

A tensor<double>(term{}) holding the document frequency that BM25 would use for each query term that searches the index field name. The document frequency is the number of documents that contain the term; it is the input to the inverse document frequency (IDF) component of BM25. This feature is exposed for debugging and for use in custom text ranking formulas such as BM25F or Bayesian BM25.

Example: for a query where term 0 and term 2 search the content field (term 1 searches a different field), queryTermDocumentFrequency(content) is:

tensor<double>(term{}):{ {term:0}:1200.0, {term:2}:57.0 }

See the reference.

Represent more complex structured data in tensors with tensorFromStructs

When converting structured attribute data into tensors, you may need to represent values across more than one dimension.

What’s new: tensorFromStructs now supports multiple mapped dimensions, letting you use multiple fields as keys to represent these relationships in a tensor.

For example, use:

tensorFromStructs(attribute(items), name, region, price, float)

to get a tensor<float>(name{},region{}) for different prices in different regions.

This function lets you represent more complex relationships in structured data directly as tensors for use in ranking.

Send Vespa telemetry directly to your observability tools with telemetry export

Telemetry export lets you ship your application’s metrics and logs from Vespa Cloud directly to your own observability backend via standard OpenTelemetry export - a fully self-service, push-based alternative to the pull-based Prometheus metrics API. You declare one or more exporters in services.xml; Vespa Cloud then pushes the selected metrics and logs you configure to the backend(s) you configure, authenticated with credentials from your vault. Configure your vault and secrets using the Vespa secret store, grant infrastructure access, add the telemetry exporter configuration, deploy, and telemetry starts flowing in minutes.

Telemetry export is available for Vespa Cloud Enclave only.

Simplify search across Chinese language variants through Asia linguistics sample apps

Vespa offers several ways to handle Chinese text — Simplified, Traditional, and the mix you typically see across zh-CN, zh-TW, and zh-HK markets. linguistics-asia compares those options side by side. It ships eight runnable sub-applications — one per integration shape — over the same dataset and the same five compare-queries, so the trade-offs are observable, not just theoretical. The basic idea is to provide a complete solution for CJK characters / terms support, and how Vespa can handle different cases with minimal custom code / logic implementation.

Other new features

Documentid indexing expression: Allows using the documentid or parts of it in the indexing language, like documentid [part] - potential uses:

• Enabling searching over parts of the document ID, e.g. namespace. Removes the need to send the value explicitly, or writing a custom document processor.
• Creating a stable and properly randomized hash value to be used as a tie-breaker for iterating over many documents.

See the reference and examples. Many thanks to Dainius Jocas (@dainiusjocas) for the contribution!

Disk size re-sampling: For users of the Vespa Enterprise Image / Vespa OSS. For environments when the disk size can change dynamically, the proton process now re-checks the disk space. The (free) size is used to block writes when the disk is close to full.,

Optimized chunked Hamming MaxSim: The existing SumMaxInvHammingFunction optimizer introduced in #32320 recognizes Hamming MaxSim over a 2D document tensor. This change adds the corresponding 3D optimizer:

query:    tensor<int8>(qt{},x[N])
document: tensor<int8>(chunk{},t{},x[N])
max(chunk, sum(qt, max(t, 1 / (1 + sum(x, hamming(query, document))))))

Many thanks to Oskari Mantere (@oskrim) for the contribution!

What’s New on YouTube

• Simplifying Your Chunking Strategy (Short video)
• The Vespa Kubernetes Operator: A Better Way to Run Vespa on K8s (Short video)
• Tensor-Native Retrieval and Ranking, Explained (Short video)
• Vespa Now: Q2 Product Update ( Webinar)
• Vespa Operator Demo

Find more videos in the @vespaai channel.

Blogs and ebooks

• Re-autoresearching MSMARCO BM25, on Vespa
• The Human Context Advantage: Takeaways from the Ai4 Keynote with Andrew Ng, Geoffrey Hinton, and Fei-Fei Li
• Your agent wants to search like a 2010 quant

Upcoming events

Vespa.ai Live: London, 9 - 10 September

Join the Vespa and retrieval community in London for a full day dedicated to building better AI search and retrieval systems. The event brings together engineers, developers, and AI practitioners to share ideas, learn from real-world deployments, and connect with others shaping the future of AI-powered search.

Learn more about Vespa Live!

it DAGENE: Trondheim, 14 September

Vespa.ai will be attending itDAGENE 2026 at Realfagbygget, NTNU in Trondheim. The event brings IT students and businesses together for two days of company stands, professional sessions, and meaningful conversations about careers and technology.

Shoptalk Fall: Nashville, 29 September - 1 October

Vespa.ai is heading to Shoptalk Fall, the essential H2 event for retail leaders shaping what comes next. Join thousands of retail changemakers in Nashville to exchange ideas, build valuable connections, and explore the innovations helping businesses stay ahead in a fast-moving retail landscape.

👉 Follow us on LinkedIn to stay in the loop on upcoming events, blog posts, and announcements.

Thanks for joining us in exploring the frontiers of AI with Vespa. Ready to take your projects to the next level? Deploy your application for free on Vespa Cloud today.

Their differing perspectives generated a lot of conversation online. They challenged each other on AI’s impact on jobs, regulation, education, and the future of technology itself.

What stood out to me wasn’t who was right. It was how their disagreements revealed the questions the AI industry is still trying to answer.

Humans Still Have the Context Advantage

As the conversation shifted from AI capabilities to education, reskilling, and the future of work, the moderator asked a practical question: 

How should we prepare people for a future where AI becomes part of nearly every job?

While Hinton warned that “routine intellectual labor is going to be done by AI,” Ng added that “humans for a long time will have a fundamental context advantage.”

At first glance, those ideas may seem contradictory. They’re not.

AI is becoming remarkably capable at routine cognitive work, but capability isn’t the same as context.

Ng explained that humans possess something today’s AI systems still lack: years of accumulated contextual knowledge. We understand our organizations, customers, business priorities, relationships, constraints, and the countless unwritten rules that shape good decisions. We know when an idea looks plausible but would never work in practice.

A model may know more facts than anyone in the room, but it still doesn’t know your business, and that distinction matters.

The future of enterprise AI won’t be determined solely by which model is the smartest. It will be determined by which systems deliver the right context at the right moment.

The Future of Work Is More Complicated Than “AI Takes Jobs”

The panel also pushed back on one of AI’s biggest headlines: that jobs will simply disappear.

The discussion wasn’t about whether AI will change work. Everyone agreed it will. The real debate is how.

Ng emphasized that jobs are changing, developers are broadening their scope, and tasks are being automated. He described how developers are already using AI to automate portions of their work and expand into broader responsibilities. Instead of specializing narrowly, they’re increasingly becoming full-stack builders capable of owning larger portions of a product. He has made this case publicly before, trading the “jobpocalypse” narrative for what he calls an “AI jobapalooza.”

Hinton’s view is that people who worry about jobs are worrying for good reason. He pointed to routine intellectual work, the kind done in call centers, as the clear near-term target. His analogy was manual labor after the mechanical digger arrived. The machine did the digging better, workers moved to other manual jobs, and offices and services absorbed them. His worry is that once AI handles routine intellectual labor, there is no obvious next place for those workers to go.

Li reframed the whole thing in layers. Every existing job is made of many tasks. Some tasks get automated, some stay untouched, and some get more of your time because the tedious parts sped up. On top of that sit three categories: jobs that transform, jobs that get created, and jobs that get genuinely displaced. Rather than asking whether AI replaces jobs, she encouraged the audience to think about how it changes the work inside those jobs.

All three perspectives can be true. Some tasks will disappear, while others will become dramatically faster, and entirely new responsibilities will emerge.

A developer may spend less time writing boilerplate code and more time designing systems. A marketer may spend less time producing first drafts and more time shaping strategy. A customer service representative may handle fewer routine requests and more complex customer conversations.

The organizations that benefit most won’t simply automate work. They’ll redesign work.

Productivity isn’t the Same as Prosperity

One of the strongest moments of the keynote came from Li: “Increased productivity does not translate to shared prosperity.”

AI is already making individuals and organizations more productive. Hinton even described a future where AI could drive “massive increases in productivity.”

But productivity is only one measure of success.

If AI allows one employee to complete five times as much work, what happens next?

Do organizations create new opportunities? Do employees take on broader responsibilities? Do customers receive better service? Or do companies simply reduce headcount?

Those are business decisions, not technological ones. The future of AI won’t be defined only by what the technology can do. It will also be shaped by how organizations choose to apply it. 

The point that stayed with me is that productivity and prosperity are not the same word. AI reliably raises productivity. Whether that turns into shared prosperity is a choice we make, not a result we are handed.

The Best AI Systems Increase Human Agency

The keynote repeatedly returned to education, but the lessons apply far beyond the classroom.

Li reminded the audience: “The core of learning is agency and motivation.”

Hinton offered an optimistic vision of AI tutors helping students learn more effectively through individualized instruction.

Despite approaching the topic from different perspectives, both speakers arrived at a similar conclusion: 

The goal isn’t to remove humans from the process. It’s to help people become more capable.

That’s true whether you’re teaching students, helping developers build software, enabling customer support teams, or assisting knowledge workers.

The best AI systems don’t replace human judgment. They amplify it.

Security Is More Nuanced Than “Open vs. Closed”

Another part of the keynote that caught my attention was the discussion around open source. It was one of the most nuanced parts of the keynote, and it was a good reminder that our industry often oversimplifies complex issues.

Hinton began by making an important distinction that is often lost in AI conversations: open source is not the same as open weights.

Open source means publishing the code so others can inspect and improve it. Open weights means releasing the trained model itself, making it easy for anyone to download, fine-tune, and adapt.

Hinton explained that he opposed releasing open weights for years because it dramatically lowered the cost of repurposing powerful models for harmful uses. But he also acknowledged that the landscape has changed. The capable open-weight models are already here. As he put it, the barrier he once worried about has largely disappeared.

Ng approached the question from a different angle. Rather than focusing on risk, he focused on competitiveness.

He argued that open models such as DeepSeek and Moonshot’s Kimi K3 are already gaining global adoption. That matters not only economically but strategically. AI models influence how people access information, solve problems, and increasingly interact with the digital world. In that sense, widespread adoption becomes a form of soft power.

His concern wasn’t that openness is inherently risky; it was that fear-driven policy could leave American open models unable to compete globally.

Then, Li argued that treating openness as a binary choice is itself the mistake.

Drawing on examples from nuclear physics and the Human Genome Project, she pointed out that society has long balanced openness and control. Scientific research can remain open while sensitive materials, infrastructure, or applications are governed differently.

Her message was simple: stop asking whether AI should be open or closed. Instead, ask what should be open, for whom, and at which layer of the technology stack.

That framing resonated with me because it’s the same kind of thinking we need across AI infrastructure.

Enterprise AI isn’t built from a single component. Models, retrieval systems, business logic, data, evaluation, and applications all have different requirements for openness, governance, and control.

My biggest takeaway here wasn’t that one speaker “won” this debate. It was that “open versus closed” is the wrong question.

The better question is: Open at which layer?

Context Is the Real Infrastructure Challenge

Listening to the discussion, I kept coming back to Ng’s observation about context. It also reinforces something I’ve seen repeatedly in enterprise AI:

Models often don’t fail because they can’t reason. They fail because they lack the context needed to reason well.

Ng described using AI to brainstorm ideas. Some suggestions were genuinely useful, while others were obviously wrong. The model generated all of them, but the human knew immediately which ones didn’t fit the situation.

That’s the difference between intelligence and understanding. 

For enterprise AI, the next challenge isn’t simply making models more intelligent. It’s giving them the context they need to make intelligent decisions.

A model can’t reliably use information that it can’t access. It can’t distinguish fresh information from stale information unless the system tells it. It can’t understand business rules, user permissions, customer history, or organizational priorities unless those signals are part of the retrieval process.

Without access to current data, business logic, organizational knowledge, and user-specific information, even the most capable model can produce an answer that is fluent, convincing (and completely wrong).

That makes context an infrastructure problem. The model may generate the response, but the system determines whether the model has the right information to generate the right response.

It’s why retrieval, ranking, freshness, personalization, and grounding matter so much.

This is exactly the problem platforms like Vespa.ai are designed to solve.

Rather than treating retrieval, ranking, inference, and serving as separate systems stitched together with multiple network hops, Vespa.ai brings them into a single query pipeline that runs computation where the data lives. AI applications can retrieve fresh information, combine lexical, semantic, and structured signals, apply business-specific ranking logic, personalize results, and perform machine-learned inference in real time. And all of this happens before the model generates a response.

As AI moves from answering questions to making decisions and taking actions, delivering the right context, not just more context, becomes the competitive advantage.

My Biggest Takeaway

The Ai4 keynote didn’t provide definitive answers about the future of AI. Instead, it showed why thoughtful disagreement is valuable.

AI may create enormous abundance, but it may also create significant disruption, and it will almost certainly transform how we work.

But the lesson I’ll remember most wasn’t about jobs, regulation, or model capabilities. It was Ng’s observation that humans retain a context advantage.

As foundation models become increasingly capable, competitive advantage won’t come from simply using AI. It will come from building systems that connect AI to the right information, the right business logic, the right user signals, and the right moment.

The model may be smart, but the system that understands your business will be the one that creates value.

AI agents need the right information to work well. Whether they manage to find it is the difference between success and failure in most real-world cases, and increasingly so as models get smarter.

The practitioners in the field of letting large language models find the information they need have passed through several stages of enlightenment.

The first stage was the vector database period of ’24, when the belief was that all you needed to do was chop text into independent chunks, create an embedding vector for each, and retrieve them via a nearest-neighbor search. This was simple, but alas, it did not work well. Chunks had too little context, and scoring based solely on vector similarity failed to reliably surface useful information.

Enter the second stage, where the learnings from human information retrieval over the last half century were combined with vector retrieval: Hybrid search, BM25, machine-learned ranking, and so on. This was a huge improvement which brought many use cases from demo stage to production quality.

But there is still vast room for improvement, and with Perplexity’s announcement of search as code, we might be officially entering the third stage. See, the search field has needed to deal with one big problem: Human users are lazy and clueless, at least when it comes to search. Given this, it turns out that the words typed into query boxes should be treated as vague indicators of what they might want, and providing any finer control is pointless, as nobody would be bothered to use it. You can see this most clearly in the evolution of Google search from letting you find web pages containing specific words to whatever it is now; thoroughly lamented but of course also deeply grounded in observation of real users.

Agents, in contrast, are not so clueless. And they certainly aren’t lazy! There’s no reason to limit their options to those of a casual human user. They should be able to:

• Search for the names of those involved near each other in text when researching a legal case
• Do a pure semantic search prioritizing high-quality sources when seeking a broad overview of a topic
• Select a year range and group by month when constructing a timeline of some events

The list goes on. Typically, an agent will string together many of these queries to reach its goal. First gaining an overview, then researching more specific topics, forming hypotheses, verifying important details in them, and so on. In short, search like an expert who knows what they are doing and really cares about the results— like a quant doing financial analysis.

It seems obvious that this will yield better results, and this is also what evaluations such as those published by Perplexity in its search as code blog post show (ignore the “code” aspect, as code execution is generally useful and where that code runs doesn’t impact quality).

Doing this in practice, with your own data, is actually quite easy. The models already know how to write complex YQL queries; they just need to be told

• That they can
• What fields are available and what they mean
• What choices they have in ranking the results

We’re working on convenience tooling for this, but how you tell them doesn’t actually matter that much; a simple textual description of what fields and rank profiles they should consider using and what they mean will do. Some example queries are also helpful. Models today are smart enough to use this effectively to connect their intents to practical YQL queries.

When creating search for humans, developers need to implement solutions that work well across a broad set of use cases, which involves making trade-offs where some types of queries cannot be improved because doing so would impact other types. When implementing for agents, the focus shifts to providing a wide toolbox for the agent to use to address their varied informational needs: broad and highly specific lexical recall, metadata attributes for filtering, grouping, and aggregation, as well as different ranking methods suited to different needs.

Accordingly, developers working on agentic search need to shift their focus from reusing techniques that have worked well for casual humans to the much richer capabilities traditionally provided by solutions for competent professionals.

It’s time to let your agents search like a 2010 quant.

Google search interest in “BM25” jumped about 5× in early August 2025. Around the same time, OpenAI’s models started volunteering BM25 noticeably more — gpt-4o named it on 12% of neutral retrieval prompts; gpt-4.1 and gpt-5-chat on 30–35%.¹ The LIMIT paper showing dense embedding models flubbing trivial retrieval landed three weeks later.²

Whatever the reasons for this spike, the renewed interest in lexical search is likely a good thing. Lexical scoring is still a very robust baseline, especially in out-of-domain or zero-shot settings. And BM25 is a great baseline, but can we do better?

Building on BM25

Earlier this month, Doug Turnbull published a really neat autoresearch experiment: let an LLM iterate on a Python BM25 reranker for 8 rounds and see how much better it gets on the MSMARCO³ passage-ranking benchmark. He randomly selects a 650k-passage slice he calls “minimarco”, and his agent is able to improve MRR@10 from 0.4913 → 0.5350 (+0.044). Very cool!

Of course, we wanted to try it for ourselves too. Our Vespa twist on it: instead of letting an LLM write arbitrary code in reranker.py, can we get a similar lift while limiting our search space to existing Vespa rank features?

Spoiler: It turns out we can indeed get a significant improvement - and one that generalises better to the full dataset too. Let’s show you exactly how we did it!

Reproducing Doug’s setup

His recipe to create the “minimarco” subset is two lines of pandas:

collection = pd.read_csv("collection.tsv", sep="\t", names=["doc_id", "description"])
minimarco = collection.sample(n=650_000, random_state=42).reset_index(drop=True)

That trims the corpus from 8.84M MSMARCO passages down to 650k. Of the 6,980 dev queries, 543 still have a labeled relevant passage that landed in our random subset (the rest are unscoreable here, since their answer isn’t in the sample).

We loaded the 650k passages into Vespa with a minimal schema, setting the two BM25 hyperparameters to the Anserini-tuned MSMARCO defaults commonly used on this benchmark.

rank-profile bm25 inherits default {
    first-phase {
        expression: bm25(description)
    }
    rank-properties {
        bm25(description).k1: 0.6      # Anserini-tuned MSMARCO defaults
        bm25(description).b: 0.62
    }
}

Running the 543 scoreable queries against this schema gives MRR@10 = 0.4907. Doug’s number is 0.4913 - close enough that the small gap could be explained by e.g. differences in stemming - the way his SearchArray’s Snowball vs. Vespa’s OpenNLP English⁴ normalize words. So we’re starting from the same baseline - now let’s see what we can add!

The three tweaks that moved the needle

We had time to try about 20 things in an afternoon. Three of them robustly survived 10 paired rotations against the running baseline — meaning each round we drew 10 different random subsets of the dev queries, evaluated both the candidate and the current best config on the same queries, and only kept changes that beat the previous best across all 10 subsets.

#1: Stopword-limit on weakAnd

Vespa’s text() operator searches its tokens with a weakAnd by default, which has a built-in document-frequency (DF) based stopword filter. We set it extremely aggressively here - ranking.matching.weakand.stopwordLimit = 0.05, which means Vespa automatically drops query terms that appear in more than 5% of docs. No need to create a hand-curated list. This makes our queries faster too - the high-DF terms have the longest posting lists, so skipping them more than halved our wall-clock latency.⁵

At 0.02 the filter has become way too aggressive and starts dropping important content words.

#2: nativeProximity

The nativeProximity feature is a continuous proximity score that rewards docs where the matched query terms are close together. Doug’s agent did something loosely similar by writing code to score based on adjacent-bigram phrase term frequencies. We just pulled it from Vespa’s rank-feature catalog:

rank-profile lexical inherits default {
    inputs {
        query(w_prox) double: 0.0      # query-time tunable weight
    }
    first-phase {
        expression: bm25(description) + query(w_prox) * nativeProximity
    }
    ...
}

Our sweep results: w_prox = 10 is a good value here, with a wide plateau (8-14 are all within noise). Wide plateaus like this are a positive sign that the gain is real, not a noise-fitted spike.

#3: fieldMatch.earliness

fieldMatch(description).earliness is a feature that rewards matches near the start of the field. A match is often a stronger signal if it appears early — writing tends to introduce its main topic up front (headline, abstract, summary). w_fm_early = 8 peaks at +0.0189 paired over the proximity-only anchor.

Iterating fast — and validating more thoroughly

A nice property of doing this in Vespa: every “did this weight help?” question is just a query parameter against the index which is already built when we fed the data. ~33 ms per query, ~0.4 s for a 109-query training eval, ~22 s to test one weight value across 10 paired rotations. It stayed fast because we weren’t paying for re-indexing or re-scoring of every query/document pair; we were just passing numbers in an HTTP body.

That speed is also what lets us validate each candidate more thoroughly. The eval set is tiny - 109 training queries per rotation - so any single split is noisy. Try 20 weight combos against one split, and one or two will look like wins by pure luck (the paired noise is ±0.014, bigger than the gains we’re chasing). Our fix is to test each candidate across many different random splits and keep only what holds up everywhere. That’s painful if every eval is an LLM call or a re-index, but nearly free when it’s just another query - so we ran 10 rotations per weight. That’s the difference between a real +0.005 and a lucky one, and possibly a big reason the single-split agent loop overfits: not enough independent looks at the data before committing to a change.

Our best recipe from this run

bm25 + 10·nativeProximity + 8·fieldMatch.earliness + sw=0.05. On the full 543-query minimarco dev set: MRR@10 = 0.5163 (+0.0256 over BM25 = 0.4907). Doug’s agent gets 0.5350 (+0.044), so it’s ahead of us by about 0.019 in absolute terms.

What happens on full MSMARCO?

First, what these numbers mean: minimarco is also where we tuned, so the minimarco column is in-sample. The 10-rotation gate guards against getting lucky on a single split — but not against fitting the 543-query subset as a whole. The full 8.84M-doc corpus is the real out-of-sample test, so that’s the column to trust for generalization.

Doug honestly flags this in his post: his agent’s gains don’t generalize well. On the full 8.84M-doc benchmark, his round-8 reranker scores 0.1991 vs BM25’s 0.1897 - only +0.0094 of the original +0.044 improvement survives. We tried our configurations on the full corpus too:

How well do our results transfer to the full dataset?

Most of our minimarco lift from both attempts survives the jump to full MSMARCO - these are all features with generalizable signal, re-tuning the weights on the full dataset might squeeze out more.

Why this gap?

• An LLM with free-form Python has more rope. By round 8 the agent hard-codes a stopword list containing vacat and medicin and a conditional like toks[1] not in ("can", "invent"). Those help on the minimarco subset; they don’t help anywhere else - it’s overfitting.
• We’re using rank features that IR researchers and Vespa engineers have already validated as carrying generalizable signal. We didn’t have to invent nativeProximity from scratch here. The LLM has to rediscover something like it from termfreq primitives within a single round, which is harder.
• The LLM writing arbitrary python does have the freedom to invent completely novel techniques, but on this extremely well-studied dataset we can perhaps consider it somewhat unlikely.

Our own “autoresearch” loop

In our results, we refer to our first experiment as “manual”, but it’s 2026 - that sweep wasn’t a human hand-typing weights either. We used a coding assistant (Claude Code) to do the exploration, with us steering at a high level and holding it to the 10-rotation rule. So really, it’s LLM-vs-LLM - what changes between the runs is the search space and how rigorously each one accepts a change, not human vs machine.

After the quick sweep we wondered: would a fully autonomous LLM agent like Doug’s, inside our constrained search space, generalize any better? We built a small loop (same eval_margin = 0.002, same rotating seeds, same gpt-5.5 model with xhigh reasoning) that edits the Vespa first-phase rank expression instead of reranker.py. The key difference from Doug’s setup: a change is accepted only if it clears the same 10-rotation paired-robustness check our manual sweep uses — so the agent and the sweep differ only in search space and autonomous-vs-steered, not in how rigorously a change is accepted. ~700 lines of Python, $6 of OpenAI spend, 30 minutes.

In our run the agent found two valuable features — nativeProximity and fieldMatch.earliness, the same pair the manual sweep landed on — and reached MRR@10 = 0.2053 on full MSMARCO (+0.0152): about 99% retention of its minimarco lift, even higher than the manual sweep’s 80% and far above the free-form-Python run’s 21%. Its absolute score sits a touch below the manual sweep’s (0.2053 vs 0.2106) because it didn’t add the stopwordLimit matching-side lever in this run — but almost all of the lift it did find carried over. That’s the constrained search space doing its job: the agent can’t encode token-specific tactics like a hard-coded stopword list, so its ceiling on overfitting is lower.

Our final config

body = {
    "yql": "select doc_id from passage where description contains ({language:'en'}text(@q))",
    "q": user_query_text,
    "ranking.profile": "lexical",
    "hits": 10,
    "language": "en",
    "input.query(w_prox)": 10.0,
    "input.query(w_fm_early)": 8.0,
    "ranking.matching.weakand.stopwordLimit": 0.05,
}

This is all that’s needed to use these features. Just a simple linear combination of well-known signals, and a demonstration that there is much more tuning potential to be had from lexical search too. And your coding agent already knows how to do it!

BM25 has served the IR community well for 30+ years, and we don’t think it’s going anywhere. But Vespa has all the pieces in place to go beyond the baseline - with lexical signals, advanced multi-vector ranking and more, and we keep working to raise the bar. Subscribe to the newsletter if you’d like to hear about it!

Going further

The full code for this experiment — the Vespa app, the paired-rotation sweep harness, and the LLM agent loop, with a step-by-step reproduction guide — is at vespaai-playground/msmarco-bm25-autoresearch.

To actually give your coding agent the Vespa knowledge to quickly succeed, there’s an official skills pack for Claude Code / Codex / Cursor / Gemini CLI: github.com/vespaai-playground/skills — which includes schema authoring, rank features, query building, etc.

Want to go even further? A weighted sum of three features is still on the simple end of the spectrum. The next step is to stop hand-weighting and let a model learn the best combination on your dataset: the RAG Blueprint collects ~190 lexical and semantic match/rank features and trains a GBDT (LightGBM) model to combine them — a more advanced version of what we did by hand here, and it mixes BM25-style lexical signals with vector-semantic ones.

Want to play with Vespa? Start with the free trial or pull the vespaengine/vespa container.

Notes

Come hang out in Vespa Slack or Discord if you want to chat ranking features or compare notes on retrieval evals. Thanks to Doug for publishing the experiment!

This month, we’re announcing several updates focused on retrieval quality, ranking flexibility, and developer productivity. Each feature is designed to help engineering teams build faster, more accurate, and more maintainable retrieval and ranking systems, while giving businesses better relevance, lower operational overhead, and more predictable performance at scale.

Let’s dive into what’s new.

Calling All Vespians: Vespa.ai Live Has Landed

As more teams build with Vespa, bringing the community closer together has become a major focus for us. Earlier this year, we ran our first virtual meetups, extending our Slack community and were joined by more than 100 Vespians from around the world — from the US and Ukraine to Singapore, Australia, Kazakhstan, Egypt, and beyond. But while virtual events are great, nothing quite compares to meeting in person — learning from peers, exchanging ideas, and continuing the conversation over coffee, beer, or wine. That’s why we’re excited to announce our first in-person community meetup: Vespa.ai Live!

The event includes technical sessions, real-world user experiences, expert panels, interactive unconference discussions, and plenty of opportunities to connect with others building in this space. Hosted by The Search Juggler, Charlie Hull, the day will bring together external experts and leading authors Trey Grainger and Doug Turnbull, Vespa engineers, community voices with speakers from Walmart, Etsy and RavenPack, and a keynote from Vespa co-founder and CEO Jon Bratseth.

On September 9, join the pre-event training, with Vespa 101: Getting started with Vespa, and Ranking 202: A deeper dive into improving retrieval quality - details and registration.

Most of all, Vespa.ai Live is intended to be community-driven — where Vespians share lessons learned and boldly go beyond the frontier of modern AI retrieval.

Learn more about Vespa Live!

Product updates

• Vespa Cloud: Detailed metric dashboards
• Vespa Cloud: Index backup
• Vespa Cloud: Fine-grained maintenance controls
• Vespa Cloud: Voyage AI, OpenAI, and Mistral AI embedding integration
• Vespa Cloud: Custom resource tags
• Vespa skills for agents
• A new query operator for text matching
• Cluster-size independent configuration of relevance effort
• Boolean array fields
• Match specific array elements
• In-memory document ids
• Search group pinning
• Near matching aware ranking
• Detect ignored write operations
• Accessing the max first phase score in re-ranking
• Geo distance in grouping

Vespa Cloud: Detailed metric dashboards

As companies deploy their large-scale latency-sensitive applications on Vespa Cloud there is a need for more detailed insights into how the application is performing. While the Vespa Cloud Console has always provided an overview metrics dashboard, many finer details have only been available to Vespa’s operations engineers.

What’s new: We have added all the metrics dashboard used by Vespa engineers to the console so that customers who want to are empowered to dig as deeply as they like. We’ve also added explanations to the dashboards to make them easier to understand.

Read more.

Vespa Cloud: Index backup

Reliability at scale means being prepared for the unexpected. Vespa Cloud now provides automated snapshot backups of indexes from content nodes, enabling catastrophe recovery without a full re-index.

What’s New: Vespa Cloud now supports backing up indexes from content nodes. The snapshot backups can be used for catastrophe restore of nodes. Read more.

Vespa Cloud: Fine-Grained maintenance controls

Vespa Cloud has always provided control over when and how application changes and Vespa upgrades are rolled out in production by the CD pipeline. In addition, Vespa Cloud does occasional OS upgrades as a background host level operation which is orchestrated but not rolled out by the CD pipeline.

We already see a need to run these processes more aggressively and anticipate that this trend will accelerate as capabilities similar to Claude Mythos become widely available. While this is necessary to maintain a strong security posture in the times ahead it has the potential to increase the impact from OS level maintenance on observed metrics, such as when these operations happen during peak traffic.

What’s new: Vespa Cloud now lets you control when these maintenance operations are allowed to happen in the same way as deployments are controlled. See the maintenance attribute in deployment.xml’s block-change tag.

Vespa Cloud: Voyage AI, OpenAI, and Mistral AI embedding integration

Embedding models are central to AI search, and it’s now simpler to use the most popular ones in Vespa Cloud.

What’s new: You can now save your API key in Vespa Cloud and invoke the embedding APIs of Voyage AI, OpenAI, and Mistral AI. These APIs can be invoked at document processing time (indexing), as well as query time. With Voyage AI and the voyage-4 model family, you can use the API for documents and a smaller, local model for queries, eliminating the need for an API call in the query path.

Read more in the Vespa embedding documentation for: Voyage AI, OpenAI, and Mistral AI.

Vespa Cloud: Custom resource tags

With Vespa Cloud Enclave, Vespa will provision resources in accounts owned by the customer. Many companies want to track these resources, e.g. for financial monitoring.

What’s New: Vespa now lets you declare custom resource tags in deployment.xml that will be applied on provisioned resources. The tag declarations can contain template variables such that resources can be tagged with e.g. the application they belong to. Read more.

Vespa skills for agents

Coding agents are good at working with Vespa applications. Giving them the relevant skills makes this even more efficient.

What’s new: We have released a collection of skills for agents working with Vespa applications, available here. This includes skills for working with application packages, feeding and queries, as well as migrating from ElasticSearch to Vespa. We run evaluations over these skills to ensure that they actually improve outcomes with current models.

A new query operator for text matching

To do lexical matching with an arbitrary input string in Vespa, you can use userInput(“my text”) in YQL. This assumes that the text can control simple syntax for controlling the matching, such as “some-field:” to specify the field to match. Sometimes, the text should just be interpreted as raw text with no such query syntax.

What’s new: Vespa now supports a new text() operator which interprets the argument text simply as raw text with no syntax. When there’s no syntax, the text can only end up searching one field or fieldSet and so the regular syntax can be used to specify the field: where my-field contains text(“my text”). Read more.

Cluster-size independent configuration of relevance effort

Vespa has various parameters to set how much effort (CPU) should be spent on providing good results in a query. These parameters are specified as a value per content node, so if you want the total expenditure to stay constant when you change the number of content nodes, you must remember to update these parameters.

This is easy to forget, and of course impossible with autoscaling activated. What’s more, when new nodes are added to clusters, they will initially have less data than other nodes, but will get the same setting as nodes with a full share of data.

What’s new: Vespa now supports alternatives of these configuration parameters prefixed by “total”, which allows you to specify values across all the content nodes. Vespa will automatically calculate the right share for each node in the cluster group, including when nodes temporarily have less data than normally.

The new “total-” parameters are:

• NearestNeighbor and WeakAnd totalTargetHits: The minimum number of hits the query operator should produce
• Match-phase total-max-hits / ranking.matchphase.totalmaxhits
• First-phase total-keep-rank-count / ranking.totalKeepRankCount
• Second-phase total-rerank-count / ranking.secondPhase.totalRerankCount

Boolean array fields

Once developers move beyond the basics to really put the power of Vespa to work they often want to pack large amounts of dynamic metadata into documents, such as for example storing information about each document’s relationship to each zip code in the US. At scale the memory efficiency with such usage really matters.

What’s new: Vespa now lets you create arrays of bits fields: field my_bits type array<bool>. Booleans can both be standalone or part of a struct type which is wrapped in an array. Read more.

Match specific array elements

Arrays in documents can be searched both as attributes and text indexes. You can also match multiple struct values or text tokens of the same array element by using the sameElement operator. In some use cases, you also want to match a specific index in the array.

What’s new: Vespa now lets you specify the array index you want to match in queries: select … where my_bits[94085] = true. You can also search for multiple indexes in the same query by using slightly more complicated syntax:

select … where my_array contains ({elementFilter:[33, 34]}sameElement(first_name contains "John", last_name contains "Doe")

See the elementFilter documentation. This is also supported in JSON queries by using an index attribute.

Search group pinning

When a content cluster has multiple groups, they will all have the same data, but their indexes will be slightly different since each node in each group has a different subset of the data written in a different order. This can lead to some inconsistency when a user is paging over a result set and hitting different groups.

What’s new: Vespa now lets you pin queries to a specific group to make pagination queries consistent. Read more.

In-memory document ids

Document ids in Vespa are only stored on disk only. This saves memory, but makes it impossible to retrieve the full ids of many documents really fast in queries and visiting.

What’s new: From Vespa 8.691 you can declare in the schema that document ids should reside in memory for fast access similar to attributes. Read more.

Near matching aware ranking

When using the near and onear query operators, the most intuitive ranking is using only the terms matching in the operator itself for rank scoring. Example:

Suppose near(term1, term2) matches document1 because of a single window where term1 and term2 appear close enough. If document1 contains term1 many additional times outside the valid proximity window, this is less relevant with respect to ranking.

What’s new: From Vespa 8.672, terms outside the match window are not considered in relevance calculations. Read more.

Detect ignored write operations

Content clusters can specify what documents they should receive in a document selection. Sending a document operation which is ignored by every cluster is not an error, but you may want to know.

What’s New: From Vespa 8.680, the document/v1 API includes a dedicated X-Vespa-Ignored-Operation HTTP response header. When an operation is ignored during routing (for example, because the target document no longer exists), this header is present and set to “true”. Read more in this issue.

Accessing the max first phase score in re-ranking

What’s new: firstPhaseMax is a new rank feature which exposes the rank score of the top scoring document locally on the node in second-phase ranking.

One usage of this is to enable dropping documents that score too low relative to the best-scoring document, by combining it with a rank-score-drop-limit.

Geo distance in grouping

What’s new: Grouping now lets you group by geo_distance:

all( group(fixedwidth(geo_distance(attribute(location), 63.4, 10.4).km, 10)) each(output(count())) )

Image Search demo

Try the image search demo to test how ranking profiles with different tensor precision in ranking affect results. Hint: binarized embeddings perform really well, read the report!

Vespa Learn

learn.vespa.ai is a self-paced course that teaches you how to build search, recommendation, and RAG applications with Vespa. You will go from zero to a working e-commerce search engine with hybrid retrieval and machine learning ranking, building it up one piece at a time across six modules.

What’s New on YouTube

• Multimodal Intelligence for Life Sciences on AWS (Webinar)
• The Personalization Problem in eCommerce AM (Webinar)
• The Personalization Problem in eCommerce EM (Webinar)
• The Relevance Problem in eCommerce (Webinar)
• Vespa Now: Q1 Product Update (Webinar)
• Zero Results Webinar

Find more videos in the @vespaai channel.

Blogs and ebooks

• Learn how Kleinanzeigen built a single system with user behavioral profiles alongside ads, WAND for fast inner-product retrieval over sparse attribute vectors, embedding-based ANN search, and click and search events processed as document updates: From Elasticsearch to Vespa: Rebuilding the Kleinanzeigen Homepage Feed.
• Scaling a Vespa Application: Feeding Fast and Furiously
• The Vespa Cloud Metrics Dashboard
• Using Large ONNX Models with External Data in Vespa Embedders
• Asymmetric Retrieval: Spend on Docs, Embed your Queries for Free
• How Metal AI Built an Agent-Driven Intelligence Platform on Vespa Cloud
• Build a High-Quality RAG App on Vespa Cloud in 15 Minutes

Upcoming events

• Commerce AI Summit London: June 3, London, UK. An executive-style event connecting retailers, brands, and AI solution providers.
• Berlin Buzzwords: June 7-9, Berlin, Germany. Europe’s leading conference for data infrastructure, search, and machine learning.
• Shoptalk Europe: June 9-11, Fira Gran Via, Barcelona. Europe’s home for retail innovation, bringing together 4,500+ trailblazers and 180+ speakers focused on AI and the future of commerce.
• Etail UK: June 16-17, Manchester, England. A leading eCommerce and retail conference focused on digital commerce strategy, customer experience, and AI-driven personalization for modern retail teams.

👉 Follow us on LinkedIn to stay in the loop on upcoming events, blog posts, and announcements.

Thanks for joining us in exploring the frontiers of AI with Vespa. Ready to take your projects to the next level? Deploy your application for free on Vespa Cloud today.

In this blog I will show you how to feed a large dataset to a Vespa Application. We will be using the full MS_marco passages dataset, which is perhaps the most comprehensive open dataset for information retrieval. It is around 4GB and contains more than 8 million passages on a wide range of topics. The goal in this blog is to show how scaling works in Vespa through feeding the entire dataset as fast as we can.

Creating the Vespa Application

We will be using a pre-made sample application as our basis for scaling but the concepts are the same for any other application.

Setup:

1. Create a tenant on Vespa Cloud:

   Go to console.vespa-cloud.com and create your tenant (unless you already have one).

2. Install the Vespa CLI using Homebrew:

   $ brew install vespa-cli
   

   Windows/No Homebrew? See the Vespa CLI page to download directly.

3. Configure the Vespa client:

   $ vespa config set target cloud
   $ vespa config set application your-tenant-name-here.scalingtutorial
   

4. Get Vespa Cloud control plane access:

   $ vespa auth login
   

   Follow the instructions from the command to authenticate.

5. Clone the sample application:

   $ vespa clone scaling-tutorial scaling-app && cd scaling-app
   

   This sample app is perfect for demonstrating scaling and performance as it is quite intensive to run both for feeding and querying. You can also check out sample-apps for other sample apps you can clone.

6. Add a certificate for data plane access to the application:

   $ vespa auth cert
   

   It is a good idea to take note of the path to the .pem files written here.

7. Add the cross-encoder and Colbert model

   Export the cross-encoder ranker model to onnx format using the Optimum library from HF or download an exported ONNX version of the model (like in this example)

   $ mkdir -p models
   $ curl -L https://huggingface.co/Xenova/ms-marco-MiniLM-L-6-v2/resolve/main/onnx/model.onnx -o models/model.onnx
   $ curl -L https://huggingface.co/Xenova/ms-marco-MiniLM-L-6-v2/raw/main/tokenizer.json -o models/tokenizer.json
   

8. Download the dataset

   The msmarco passages dataset can be found here. Download, unzip it and place it in the ext/ folder in our application.

   NOTE: You will need around 8GB of free disk space for the dataset and the subsets we will be creating.

9. Prepare the dataset for Vespa

   Then run the script to convert it into the vespa feed format:

   $ python3 ext/transform_ms_marco.py
   

   which gives us the dataset and a few subsets of various sizes to feed to our application.

Deploying and Feeding

We now have everything we need for deployment, feeding and scaling! Scaling a vespa application is largely managed through the services.xml file. This is what the file currently looks like:

<?xml version="1.0" encoding="utf-8" ?>
<!-- Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -->
<services version="1.0" xmlns:deploy="vespa" xmlns:preprocess="properties" minimum-required-vespa-version="8.311.28">

  <container id="default" version="1.0">

    <nodes deploy:environment="dev" count="1">
      <resources vcpu="1.0" memory="8Gb" architecture="arm64" storage-type="local" disk="59Gb"/>
    </nodes>
   
    <search/>
    <document-api/>

     <!-- See https://docs.vespa.ai/en/embedding.html#huggingface-embedder -->
    <component id="e5_embedding_model" type="hugging-face-embedder">
            <transformer-model url="https://huggingface.co/intfloat/e5-small-v2/resolve/main/model.onnx"/>
            <tokenizer-model url="https://huggingface.co/intfloat/e5-small-v2/raw/main/tokenizer.json"/>
            <prepend>
                <query>query:</query>
                <document>passage:</document>
            </prepend>
    </component>

    <!-- See https://docs.vespa.ai/en/embedding.html#colbert-embedder -->
    <component id="colbert_embedding_model" type="colbert-embedder">
      <transformer-model url="https://huggingface.co/colbert-ir/colbertv2.0/resolve/main/model.onnx"/>
      <tokenizer-model url="https://huggingface.co/colbert-ir/colbertv2.0/raw/main/tokenizer.json"/>
    </component>

     <!-- See https://docs.vespa.ai/en/reference/embedding-reference.html#huggingface-tokenizer-embedder-->
    <component id="tokenizer" type="hugging-face-tokenizer">
      <model path="models/tokenizer.json"/>
    </component>

  </container>

  <content id="msmarco" version="1.0">
    <min-redundancy>1</min-redundancy>
    <documents>
      <document mode="index" type="passage"/>
    </documents>
    <nodes deploy:environment="dev" count="1">
      <resources vcpu="1.0" memory="8Gb" architecture="arm64" storage-type="local" disk="59Gb"/>
    </nodes> 
    <engine>
      <proton>
        <tuning>
          <searchnode>
            <requestthreads>
              <persearch>4</persearch>
            </requestthreads>
            <feeding>
              <concurrency>1.0</concurrency>
            </feeding>
          </searchnode>
        </tuning>
      </proton>
    </engine>
  </content>

</services>

The important parts to take note of in this tutorial are the two resource specifiers in the <container> and <content> tags:

Container

<nodes deploy:environment="dev" count="1">
    <resources vcpu="1.0" memory="8Gb" architecture="arm64" storage-type="local" disk="59Gb"/>
</nodes>

Content

<nodes deploy:environment="dev" count="1">
    <resources vcpu="1.0" memory="8Gb" architecture="arm64" storage-type="local" disk="59Gb"/>
</nodes>

This is where we configure the machine resources that our Vespa application runs on in Vespa Cloud.

NOTE: when deploying to dev we need to add the <nodes deploy:environment="dev"> specifier to ensure we actually get the resources we ask for, otherwise we default to what is quickly available.

Adding more resources or more nodes are the main parameters that need to be tweaked in order to scale your application. Right now we have provisioned the smallest amount of resources to our application.

Deploy the application to Vespa Cloud:

$ vespa deploy --wait 900

(It might take a little bit of time for all services and nodes to go up and start running.)

You can follow the progress of the deployment from the terminal or in your tenant in your cloud console. When it is finished you should get the message:

Application up!

If you go to your cloud console you should be able to see your application. Note that we haven’t fed it any documents yet, so it should look something like this:

Let’s feed some documents. Feed the smallest dataset to Vespa using:

vespa feed ext/corpus_transformed_1000.jsonl

or, on Unix systems:

time vespa feed ext/corpus_transformed_1000.jsonl

to see how long it takes.

It will take a few minutes as we are doing heavy computations on very modest resources.

If you want to see a live count of how many documents that are in Vespa you can run:

vespa query 'yql=select * from passage where true' 'hits=0' 'ranking=unranked'

to see how many documents have been processed so far (under documents).

On this lowest resource configuration we get this result.

vespa feed ext/corpus_transformed_1000.jsonl  4.96s user 7.11s system 3% cpu 5:56.05 total 

If we were to try and feed the whole 8.8 million passage msmarco dataset on this instance it would take more than a month to finish feeding!

We can do better!

Scaling

Before scaling the application we’ll delete the documents from our instance so that we have a fresh start.

Now lets assign more resources to our Vespa instance. From our schema we see that we are doing extensive computations during feeding (notice the configuration in the indexing parameters)

Schema

  field e5_embedding type tensor<bfloat16>(x[384]) {
    # Using the e5 embedding model defined in services.xml
    indexing: input text | embed e5_embedding_model | attribute | index
    attribute {
      distance-metric: angular
    }
    index { # override default hnsw settings 
      hnsw {
        max-links-per-node: 32
        neighbors-to-explore-at-insert: 400
      } 
    }
  }

  field colbert_embeddings type tensor<int8>(dt{}, x[16]) {
    # No index - used for ranking, not retrieval 
    indexing: input text | embed colbert_embedding_model | attribute
    attribute: paged
  }

Embedding in Vespa happens in the container cluster, so it is a very reasonable guess that if we can make the embeddings go faster, our whole system will be faster (bellow in this blog we will show how to more thouroughly deduce scaling parameters). So lets start by scaling up the resources for the container node. To see what resource configurations we have available we must look at the instance type page in the documentation. Embedding-computations are best suited to run on GPUs, so we will select an instance type with a GPU:

Container

<nodes deploy:environment="dev" count="1">
    <resources vcpu="4.0" memory="16Gb" architecture="x86_64" storage-type="local" disk="125Gb">
        <gpu count="1" memory="16.0Gb"/>
    </resources>
</nodes>

Replace the resources in the container node in services.xml with the new instance type (see above). Leave the content node resources as is for now.

Run the command for checking document count to make sure that it is zero:

vespa query 'yql=select * from passage where true' 'hits=0' 'ranking=unranked'

and redeploy:

$ vespa deploy --wait 900

When the deployment is finished we’ll time the feeding process again.

time vespa feed ext/corpus_transformed_1000.jsonl

vespa feed ext/corpus_transformed_1000.jsonl  0.40s user 0.53s system 8% cpu 11.371 total 

11.4 seconds, that’s more like it! Instead of a month, this new instance would be able to crunch through the full dataset in just around a day!

We have now significantly upgraded a part of the hardware Vespa is running on. But before we scale up further we shall take a look at the metrics tab for our application. Go to Metrics and then resources.

This is where you see the resource usage history in your vespa instance, but most importantly it gives you a clear image of where your application is bottlenecked. The bottleneck for your application will be different depending on how your application is configured and the kind of computations you do. The previous 1000-line dataset was no match for the upgraded instance, so lets give it a bigger one to get some proper bottleneck data:

Delete the documents from the instance again, wait a bit, and run the command to ensure that we have no documents in our application

vespa query 'yql=select * from passage where true' 'hits=0' 'ranking=unranked'

Now we’ll feed the 50 000 line dataset to properly test and time the upgraded instance.

time vespa feed ext/corpus_transformed_50000.jsonl

Result

vespa feed ext/corpus_transformed_50000.jsonl  17.44s user 22.68s system 11% cpu 5:56.41 total (~17.6 hours for full dataset)

This is a more accurate reading of the instance’s performance, and at 5min 56s to feed 50 000 documents, the full dataset would take around 17 and a half hours.

Look at the resources in the metrics and set it to show only the last 30 minutes so that we can see more clearly what went on. Notice the CPU-utilisation and the GPU-utilisation graphs. Notice that the GPU usage on the container node hit 100% and stayed there for the entire feeding process. The CPU usage on the container node peaked at 80% but leveled at around 60% and the content node’s CPU barely went over 50%.

It is clear that on this Vespa Instance, the bottleneck for better feeding performance lies in the GPU processing. If we want to improve the feeding performance of the system, then we must increase the amount of GPUs in the container node.

Now that we know where the problem lies: Lets make it go faster! We’ll increase the amount of GPU nodes to 5 with the count="5" parameter in the container node in services.xml.

Container

<nodes deploy:environment="dev" count="5">
    <resources vcpu="4.0" memory="16Gb" architecture="x86_64" storage-type="local" disk="125Gb">
        <gpu count="1" memory="16.0Gb"/>
    </resources>
</nodes>

save the services.xml file and redeploy:

$ vespa deploy --wait 900

Now lets feed the larger dataset:

time vespa feed ext/corpus_transformed_500000.jsonl

Result

vespa feed ext/corpus_transformed_500000.jsonl  97.59s user 95.22s system 10% cpu 29:38.18 total (~8.8 hours for full dataset)

If we extrapolate the results we see we got around twice the speed of the single-container node instance. But why not 5 times the speed? Let’s look at the metrics.

We see that the container GPU utilization now sits comfortably at around 50% and the container CPU at around 20-30%. But the content node CPU utilization sits near 100%. The 5 content nodes with GPUs saturate the single content node’s ability to take in data. We have found the new bottleneck of the system.

We’ll add some more content nodes:

Content

<nodes deploy:environment="dev" count="2">
    <resources vcpu="1.0" memory="8Gb" architecture="arm64" storage-type="local" disk="59Gb"/>
</nodes> 

Delete the documents, redeploy, and refeed:

$ vespa deploy --wait 900

time vespa feed ext/corpus_transformed_500000.jsonl

Result

vespa feed ext/corpus_transformed_500000.jsonl  92.22s user 88.21s system 17% cpu 16:51.51 total (~5.0 hours for full dataset)

Adding the second content node almost doubles the performance again. Look at the metrics to see what is going on.

We see now that the container GPU (70-80%) and the content node CPU (80-90%) are both highly utilised, whilst the container node CPU is around 40%. Since we are already on the smallest instance type with a GPU we can’t scale down the cpu to match the others, so we have actually found a near optimal balance of container and content node resources for feeding this application.

Now that we have found a good balance, lets really scale up!

Feeding Fast: 20 GPUs

If we want serious feed throughput, we need serious hardware. Let’s scale the container and content nodes proportionately and jump to 20 GPU container nodes and 8 content nodes at the same time:

Container

<nodes deploy:environment="dev" count="20">
    <resources vcpu="4.0" memory="16Gb" architecture="x86_64" storage-type="local" disk="125Gb">
        <gpu count="1" memory="16.0Gb"/>
    </resources>
</nodes>

Content

<nodes deploy:environment="dev" count="8">
    <resources vcpu="1.0" memory="8Gb" architecture="arm64" storage-type="local" disk="59Gb"/>
</nodes>

NOTE: At this point you will most likely hit the quotaExceeded error when you try to deploy. Vespa Cloud tenants have a default quota that prevents you from accidentally spending a lot of money. If you want to go past it, reach out to Vespa support. With the limit raised, redeploy:

$ vespa deploy --wait 900

Delete any existing documents, wait for the count to hit zero, and feed the 500 000 line dataset again:

time vespa feed ext/corpus_transformed_500000.jsonl

Result

vespa feed ext/corpus_transformed_500000.jsonl  59.66s user 48.08s system 42% cpu 4:13.19 total (~1 hour 15 min for full dataset)

At an estimated 1 hour and 15 minutes for the full dataset we see that we got pretty much exactly 4x feeding speed with 4x the resources. We also see that the utilisation metrics are essentially the same as the last run (feeding at 11:30), just faster.

Feeding Furiously: 100 GPUs

Finally, because we can: 100 GPU container nodes and 40 content nodes, and this time we will feed the full 8.8 million passage dataset in one go.

Container

<nodes deploy:environment="dev" count="100">
    <resources vcpu="4.0" memory="16Gb" architecture="x86_64" storage-type="local" disk="125Gb">
        <gpu count="1" memory="16.0Gb"/>
    </resources>
</nodes>

Content

<nodes deploy:environment="dev" count="40">
    <resources vcpu="1.0" memory="8Gb" architecture="arm64" storage-type="local" disk="59Gb"/>
</nodes>

Delete the documents, Deploy, then feed the full dataset:

time vespa feed ext/corpus_transformed_full.jsonl

Result

vespa feed ext/corpus_transformed_full.jsonl  695.31s user 605.48s system 108% cpu 20:04.23 total

The Vespa instance managed to process more than 8.8 million passages, with embeddings and ColBERT vectors computed for every single one, in just over 20 minutes (over a fast internet connection).

The feed client also gives us a nice summary at the end:

{
  "feeder.operation.count": 8841823,
  "feeder.seconds": 1201.608,
  "feeder.ok.count": 8841823,
  "feeder.ok.rate": 7358.324,
  "feeder.error.count": 399,
  "http.request.count": 8844266,
  "http.response.latency.millis.avg": 167,
  "http.response.code.counts": {
    "200": 8841823,
    "429": 2044
  }
}

The feeding process had an average feed rate of around 7358 documents per second. Now that is fast and furious!

Conclusion

The best way to scale your Vespa instance is to use the metrics dashboards to see where the bottlenecks lie. There is no singular best instance of Vespa as the computational requirements are highly dependent on how you define your application. Feed the instance a sizable corpus to see how it performs under sustained load, and adjust its resources accordingly.

The Vespa Cloud metrics dashboard is designed for exactly that. Instead of treating monitoring as a wall of graphs, it helps you move from symptom → bottleneck → action.

Start with three questions

Most production issues can be reduced to three questions:

1. Is the system healthy?
2. Where is latency added?
3. Are we running out of resources?

The dashboard mirrors that flow.

1. Is the system healthy?

Start on the Overview tab. This is the fastest place to answer “is anything obviously broken?”. A healthy system keeps read and write QoS close to 100%. If it drops, look at whether 4xx or 5xx responses are rising — 5xx responses usually mean the problem is on the server side. A rise in degraded or failed queries means it is time to continue into the Query tab.

See the docs for the full reference: Metrics Overview tab.

2. Where is latency added?

Latency in Vespa is layered — a slow request is not just “slow”, it can be slow in different parts of the path:

HTTP → container → content nodes → ranking

That is why the dashboard shows several latency metrics for what feels like the same request. If HTTP latency is much higher than query latency, payload size or network overhead may be the issue. If search-protocol latency on the content nodes is high, the bottleneck is deeper in the system.

See the docs for a layer-by-layer walkthrough: Query tab and Feed tab.

3. Are we running out of resources?

Once you know where the slowdown is, switch to the Resources tab. As a rule of thumb, sustained utilization above roughly 80% is a sign the cluster may need more headroom. If one host is much hotter than the others, enable per-host metrics and look for uneven load distribution.

See the docs for healthy-value tables and scaling guidance: Resources tab.

What’s new in the latest revision

The dashboard has picked up a few improvements worth calling out.

Health Indicators on the Overview tab

The Overview tab now opens with a dedicated Health Indicators row — five stat panels that surface stability issues in a single glance: Core Dumps (1h), Restarts (1h), Feed Blocked, Content Cluster with Groups/Nodes Down, and Container Nodes with Services Down.

Details and healthy values: Health Indicators.

Annotations for Service restart and Core dump

Annotations are the vertical lines drawn across every chart when an operational event happens — Vespa upgrades, feed blocked, data migration, reindexing, autoscaling changes. Two annotations were added recently and they are worth flagging:

• Service restart — fires when a Vespa service process restarts. Outside of planned upgrades, restarts usually mean a crash, OOM, or forced stop.
• Core dump — fires when a process core-dumps. Should be extremely rare.

When a latency anomaly lines up with one of these annotations, you get the context for the change without having to infer it from the graph alone. Both signals also feed the Overview’s Health Indicators row, so the same event shows up in three places: the counter, the annotation line, and the Health tab’s historical time series.

Full annotation reference: Annotations.

Container thread pool rows, one per configuration case

The Resources tab used to have a single thread-pool row that was mostly empty — a container only has the thread pools that match its services.xml configuration (<search>, <document-api>, or both). The row has been split into three case-specific rows:

• Thread Pools (search + document-api) for full-feature containers
• Thread Pools (search only) for query-only containers
• Thread Pools (document-api only) for feed-only containers

Classification is automatic — hidden variables derive the cluster list per case from Prometheus set operations, so only relevant rows render for a given deployment. Each thread pool now gets its own panel with avg (green) and max (yellow dashed) on the same chart.

Details: Container Thread Pools.

JVM memory breakdown (heap / direct / native)

The Resources tab separates the three layers of container memory: heap, direct, and native. This matters on container nodes that run embedders or local LLM components — model weights are memory-mapped and partially resident, but KV cache and compute buffers are allocated upfront as native memory. When node memory is high but heap and direct look normal, the native layer is usually where to look.

Details: JVM memory breakdown.

A simple workflow

A practical way to use the dashboard during an incident:

1. Open Overview and scan the Health Indicators row.
2. Confirm the symptom (QoS drop, latency spike, error-rate increase).
3. Use Query or Feed to find the slow layer.
4. Use Resources to confirm whether the cluster is saturated.
5. Cross-reference annotations for restarts, upgrades, reindexing, or migration.

That flow gets from “latency is up” to “this is the actual bottleneck” much faster than scanning every chart. The common workflows section of the docs has recipes for the most frequent scenarios.

Summary

The Vespa Cloud metrics dashboard works best as a troubleshooting tool — not a metrics catalog. Start with health, follow the latency path, confirm with resources, and use annotations to connect spikes to real events. The tab reference, healthy-value tables, and step-by-step workflows live in the Monitoring documentation.

This was a long requested feature on our tracker (see GitHub issue #28761).

The 2 GB limitation

ONNX uses Google’s Protocol Buffers as its serialization format. Protobuf has a hard limit of 2 GB on message size. For smaller models, this is not a problem — all tensor data (the model weights) is embedded directly in the .onnx file, making it self-contained.

As models grow larger, they inevitably hit this limitation. For a model exceeding 2 GB, ONNX tooling splits it into two parts:

• A small .onnx file containing the model graph structure (typically a few hundred KB to a few MB).
• One or more external data files (commonly named .onnx_data) containing the actual tensor weights.

Note that reduced-precision variants of these models (INT8, FP16, etc.) are often small enough to fit in a single self-contained .onnx file. The external data split primarily affects the full-precision versions.

Previously, if you pointed a Vespa embedder at a model with external data files, ONNX Runtime would fail to load it because the data files were not available alongside the model file.

What changed

Vespa embedders now automatically handle ONNX models with external data files. When you configure an embedder with a URL pointing to an .onnx file, Vespa inspects the model to check whether it references any external data files. If it does, Vespa downloads those files automatically before loading the model.

This feature is available starting from Vespa 8.544.

How to use it

Here is an example using EmbeddingGemma 300M, which uses external data:

<container id="default" version="1.0">
  <component id="gemma" type="hugging-face-embedder">
    <transformer-model
      url="https://huggingface.co/onnx-community/embeddinggemma-300m-ONNX/resolve/main/onnx/model.onnx"/>
    <tokenizer-model
      url="https://huggingface.co/onnx-community/embeddinggemma-300m-ONNX/resolve/main/tokenizer.json"/>
    <max-tokens>2048</max-tokens>
    <prepend>
      <query>task: search result | query: </query>
      <document>title: none | text: </document>
    </prepend>
  </component>
</container>

If you are deploying to Vespa Cloud, you can also use models from the Vespa Model Hub that use external data. For example, the Multilingual-E5-large model (will be available on Vespa Cloud 8.668+):

<container id="default" version="1.0">
  <component id="e5" type="hugging-face-embedder">
    <transformer-model model-id="multilingual-e5-large"/>
    <max-tokens>512</max-tokens>
    <prepend>
      <query>query: </query>
      <document>passage: </document>
    </prepend>
  </component>
</container>

This works with our ONNX-based embedders:

• hugging-face-embedder
• colbert-embedder
• splade-embedder

It’s also possible to use private models — authentication tokens are propagated when downloading external data files.

Current limitations

There are a few constraints to be aware of:

• Embedders only. Models used directly in ranking expressions must still be self-contained and under 2 GB.

• URL-referenced or Model Hub models only. Models bundled in the application package using the path attribute do not support external data. Models referenced via url or model-id (Vespa Cloud) are supported.

• External data files must be co-located with the model. The external data files are resolved relative to the model URL. They must be in the same directory (or a subdirectory) as the .onnx file.

See the ONNX model documentation for the full list of requirements.

If you need more extensive support for ONNX models with external data — for example in ranking expressions — feel free to file an issue.