Qdrant Publishes a 10B-Vector Benchmark and Open Tools to Rerun It

Qdrant has published a benchmark meant to move vector-database testing beyond small collections, along with the open-source software needed to rebuild and run the test. FineWeb-10B contains 10.07 billion dense vectors and 10.07 billion sparse vectors, while Supernova generates embeddings, calculates exact answers, loads databases and runs evaluations. The release makes the machinery inspectable; it does not by itself make a vendor-built comparison independent.

The test is built for deep retrieval

Qdrant released FineWeb-10B and Supernova on September 1. The corpus was created from Hugging Face’s FineWeb data with the gte-multilingual-base model, and contains 24.47 TB of vector data plus 28.66 TB of source text and metadata. Those files are substantial enough to make storage and ingestion part of an evaluation, rather than treating search as an isolated query problem.

The benchmark supplies exact top-1,000 ground truth: the brute-force nearest-neighbor results against which a database’s search output can be measured. Qdrant says it computed those results for 120,000 dense, sparse and filtered queries, requiring more than one quadrillion distance calculations across the corpus.

Supernova makes the method portable

Supernova turns the dataset from a download into a process. Its four stages cover embedding generation, brute-force ground-truth calculation, database loading and evaluation benchmarking. YAML configuration files define jobs, while its distributed tooling can run the pipeline across cloud and high-performance computing environments.

The evaluation can track

• Ingestion throughput and build times, testing how a system handles loading the corpus.
• Queries per second, recall against the supplied ground truth, and latency at the p50, p95 and p99 percentiles.
• Qdrant, Milvus and Elasticsearch through Supernova’s loading and stress-testing components.

Reproducible is not yet independent

Qdrant created the benchmark, chose its initial representations and wrote the evaluation framework. Publishing code and support for competing systems gives other operators a way to inspect the workload and run it on their own infrastructure. But the first benchmark design remains Qdrant’s; equivalent results from teams operating Milvus or Elasticsearch would be the meaningful test of whether it becomes a shared comparison point.

That distinction matters most for buyers weighing systems at large scale. FineWeb-10B can expose more of the workload, the measurements and the ground truth than an opaque performance claim. Its status as a practical standard will depend on published reruns that make database settings and operating choices equally visible.