Open Source

Vespa Product Updates, October 2020: New Container Thread Pool Configurations & Improved Document/v1 API Throughput

Kristian Aune, Tech Product Manager, Verizon Media

Developers using their computers to create software

Largely developed by Yahoo engineers, Vespa is an open source big data processing and serving engine. It’s in use by many products, such as Yahoo News, Yahoo Sports, Yahoo Finance, and the Verizon Media Ad Platform. Thanks to feedback and contributions from the community, Vespa continues to grow.

In the previous update, we mentioned ONNX runtime integration and feeding improvements. This month, we’re excited to share the following updates:

New Container Thread Pool Configurations

When deploying, application changes are live reloaded into the running JVM. New code requires JVM JIT compilation, which temporarily loads the container and causes increased query latencies for a second or two. Many parallel threads aggravate this problem. Vespa now has a dedicated container thread pool for feeding. Compared to the previous default of a fixed size with 500 threads, it now defaults to 2x logical CPUs. This both improves feed throughput and reduces latency impact during deployments.

Improved Document/v1 API Throughput

Vespa users feed their applications through feed containers in their Vespa cluster, using either an asynchronous or a synchronous HTTP API. Optimizations and fine-tuning of concurrent execution in these feed containers, and the change to asynchronous handling of requests in the synchronous document/v1 API, has made the feed container more effective. This has greatly increased quality of service for both search and feed during container restarts. As a bonus, we also see a 50% increase in throughput for our performance test suite of the synchronous HTTP API, since Vespa version 7.304.50 onwards.

Visibility-delay for Feeding No Longer Needed

Visibility-delay was used to batch writes for increased write throughput. With the recent optimizations, there is no gain in batching writes as it’s now just as fast without it. Due to this, the batch code has been removed. Visibility-delay is still working for queries, with a short cache with max 1 second TTL. We recommend not using this feature anymore, as there is no longer an advantage to having this delay.


We welcome your contributions and feedback (tweet or email) about any of these new features or future improvements you’d like to request.