1. The Memory Wall of In-Memory Vector Search
The gold standard algorithm for Approximate Nearest Neighbor (ANN) search has long been HNSW (Hierarchical Navigable Small World). However, HNSW requires storing all high-dimensional vector embeddings and graph index pointers entirely in RAM. Storing one billion 1536-dimensional vectors in RAM requires over 6 Terabytes of memory, costing tens of thousands of dollars per month in enterprise cloud hosting.
2. Vector Quantization Techniques: Scalar and Product Quantization
Vector Quantization slashes memory footprints by compressing floating-point values: Scalar Quantization (SQ8) converts 32-bit floats into 8-bit integers, immediately reducing memory requirements by 75% with negligible recall loss. Product Quantization (PQ) divides vectors into smaller sub-vectors and maps them to shared centroid codebooks, achieving up to 95% memory compression.
3. DiskANN: Graph Search on Fast NVMe Solid State Drives
Developed by Microsoft Research, DiskANN shatters the RAM memory wall by designing graph structures optimized specifically for modern NVMe SSDs. DiskANN keeps only a compressed index in RAM to guide the initial search, storing full-precision vectors on cheap NVMe SSDs and retrieving them using parallel asynchronous I/O (io_uring). This enables a single workstation with 64GB of RAM to query one billion vectors with sub-10ms latency.
4. Production Deployment Recommendations
For datasets under 10 million vectors, in-memory HNSW remains optimal for lowest latency. When scaling beyond 100 million vectors, transitioning to DiskANN-powered engines (such as Milvus or Qdrant with disk storage enabled) slashes infrastructure operational costs by over 80%.
