Doubling Query Embedding Throughput with Dynamic Batching in Vespa Cloud

Introduction
As query volumes increase, generating embeddings can become a bottleneck for semantic search, since every query passes through an embedding model. Processing requests individually uses the GPU inefficiently, which limits throughput and increases latency under heavy load.
Vespa 8.784.16 brings a new inference runtime for container nodes in Vespa Cloud, built on NVIDIA Triton Inference Server. Its main benefit is dynamic batching, which combines concurrent inference requests into batches to use the GPU more efficiently. It also limits how many inference calls run at once, which reduces contention and memory use.
Our benchmarks show that dynamic batching more than doubles query embedding throughput on a GPU, while adding just a few milliseconds of latency.
We have also added instance types with NVIDIA L4 and L40S GPUs, faster than the existing T4, described at the end of this post. In the rest of the post, we explain how concurrency and dynamic batching work, show how to configure them in Vespa, and look at the benchmark results.
Inference queue, concurrency, and batching
Vespa runs machine-learned models on both content nodes and container nodes. Content nodes run ranking models close to the data, while container nodes run embedding models and heavier ranking models such as cross-encoders. Low-latency embedding often requires container nodes with a GPU.
Open-source Vespa, and Vespa Cloud until now, run each embedding request as a separate inference call. More requests in flight means more calls running at the same time. Each call has overhead for launching operations and transferring data, and uses memory for intermediate results. Many concurrent calls therefore increase memory use and GPU contention.
With the new inference runtime, Vespa Cloud puts requests in an inference queue. The concurrency setting controls how many inference calls can run at once. Requests wait in the queue until a call can start. This bounds the number of active calls, which limits GPU memory use and contention.
Embedding a single short text is too little work to keep a GPU busy. A batch groups several texts in one inference call. This gives the GPU more work per call and spreads the overhead across requests, so a batch of short texts costs little more than a single text.
With dynamic batching, Vespa forms each batch from the requests already in the queue, up to a configured maximum batch size. It adds no delay to wait for more requests: at low load, a batch may contain just one request, while at higher load, more requests are waiting and batches grow. This is why the concurrency limit matters. Without it, every request would start its own inference call right away, nothing would ever wait in the queue, and there would be nothing to batch.
Requests can only share a batch when their tokenized inputs have the same length. Vespa applies padding to extend inputs to one of a few configured lengths, called padding buckets. For example, with buckets of 32 and 64 tokens, inputs of 12 and 25 tokens are both padded to 32 and can be in the same batch. An input of 40 tokens is padded to 64 and runs in a separate batch. Inputs longer than 64 tokens are not padded and can only share a batch with inputs of exactly the same length.
Each batch runs as one inference call, so the concurrency limit applies to batches. The diagram below shows dynamic batching with a concurrency of two and a maximum batch size of eight. Requests C, D and E arrive while A and B are running, and wait in the queue. When A finishes, C, D and E start as one batch of three while B keeps running. Vespa does not wait for the batch to fill up to eight.
Configuring batching and concurrency in Vespa
Batching and concurrency are configured in the embedder component in services.xml.
The example below uses Arctic Embed M v2.0 on a node with an NVIDIA L4 GPU.
<container id="default" version="1.0">
<component id="arctic" type="hugging-face-embedder">
<transformer-model model-id="snowflake-arctic-embed-m-v2.0"/>
<max-tokens>512</max-tokens>
<transformer-output>token_embeddings</transformer-output>
<pooling-strategy>cls</pooling-strategy>
<normalize>true</normalize>
<prepend>
<query>query: </query>
</prepend>
<batching max-size="8" max-delay="0ms" padding-buckets="32 64"/>
<concurrency type="absolute">1</concurrency>
</component>
<search/>
<document-api/>
<nodes count="1">
<resources vcpu="4" memory="16Gb" disk="250Gb">
<gpu count="1" memory="24Gb"/>
</resources>
</nodes>
</container>
The settings are:
max-size: the maximum number of requests in a batch.max-delay: how long to wait for a batch to fill, keep it at0ms.padding-buckets: the token lengths that inputs are padded to.concurrency: how many inference calls (batches) can run at once.
You can use this configuration as a starting point for embedding short queries on a GPU.
For documents and chunks, adding larger buckets may also help. However, a long input can keep the GPU busy on its own, leaving little benefit from batching. The input length at which this happens depends on the model and GPU.
On CPU nodes, keep the defaults for batching and concurrency, and do not configure padding buckets. The CPU cores already run separate requests in parallel, and padding only adds work. Concurrency defaults to the number of vCPUs, so concurrent inference calls keep all cores busy under load.
The configuration reference covers defaults, relative concurrency, and the requirements for padding. See padding troubleshooting to check whether padding is active for your model, and tuning local model inference for more advice on choosing the settings.
Benchmarks and results
We used a Vespa Cloud node with 4 vCPUs, 16 GB of memory and an NVIDIA L4 GPU with 24 GB (AWS g6.xlarge), the same as in the example configuration above.
The Vespa application indexes 2000 documents from the NFCorpus dataset.
The corpus is small on purpose, so that query embedding, not search, dominates the cost of a query.
We used queries from the same dataset, which have about 15 tokens on average.
The application uses hybrid search to combine BM25 and nearest neighbor search over HNSW,
with reciprocal rank fusion to merge the results.
We chose two modern embedding models from the Vespa Model Hub: EmbeddingGemma 2 and Arctic Embed M v2.0.
Both models run in full precision with 256 output dimensions.
The baseline uses the previous inference runtime, where each query runs in a separate inference call without a concurrency limit.
With dynamic batching, we use max-size 8, padding buckets 32 64, and concurrency 1.
The load increases in steps of 50 queries per second until latency rises sharply. The measured query latency includes queueing, embedding, search, and result fetching. It excludes response rendering and the network round trip between the client and Vespa. We measure each point for 60 seconds and report the median of up to three runs.

The table below shows the highest load before median latency rises sharply without dynamic batching, and twice that load with dynamic batching.
| Model | Dynamic batching | Throughput | Median latency | p99 latency |
|---|---|---|---|---|
| EmbeddingGemma 2 | Disabled | ~300 q/s | 15 ms | 303 ms |
| EmbeddingGemma 2 | Enabled | ~600 q/s | 15 ms | 148 ms |
| Arctic Embed M v2.0 | Disabled | ~450 q/s | 10 ms | 88 ms |
| Arctic Embed M v2.0 | Enabled | ~900 q/s | 11 ms | 113 ms |
The main results:
- Dynamic batching doubles the throughput without a significant increase in median latency.
- Peak throughput is about 2.5 times higher with dynamic batching.
- For all but low loads, p99 latency is lower with dynamic batching.
- Without batching, latency rises sharply once the node can no longer keep up with the query load. With dynamic batching, it increases more gradually.
New GPU instance types
In addition to the NVIDIA T4 (16 GB), Vespa Cloud now offers instance types with newer and faster GPUs:
The L4, which we used for the benchmarks in this post, is a good fit for embedding queries and short documents. The L40S has twice the memory and more compute, for larger models or heavy document embedding.
Conclusion
Dynamic batching lets a GPU embed several texts in one inference call instead of one at a time. In our benchmarks on an NVIDIA L4, the new inference runtime with dynamic batching served twice the query load at the same median latency, so fewer GPU nodes are needed for the same query rate. It is available on container nodes in Vespa Cloud from Vespa 8.784.16. To try it, start with the configuration above, then measure throughput and latency in your own application, following the tuning guide.