How Vector Databases Work Internally: Architecture, Storage and Distribution

Most engineers who use a vector database think of it as a black box: send in a vector, get back the nearest neighbors. That model works until something goes wrong. A search that should return ten results returns three. Ingestion slows down at two million vectors. A metadata filter changes the answer set unexpectedly. Latency spikes under concurrent load.

Every one of those failure modes has an architectural explanation. Understanding the write path, storage model, segment lifecycle, maintenance operations, and distributed topology gives you the context to identify the responsible layer.

This pillar focuses on the internal architecture of a production vector database: ingestion, persistence, segments, compaction, CRUD operations, sharding, replication, and disaggregated deployment. Search algorithms, metrics, filtering, and request execution appear only as architectural components and link to dedicated technical articles for their mechanics and tuning.

If you are new to the field, the Vector Database Fundamentals series covers what a vector database is, what embeddings are, and why the underlying indexes work the way they do. This series assumes that foundation and goes a level deeper into the internal machinery.

The Four Layers of a Vector Database

The write path prepares indexed vectors; the query path embeds a query and searches the same index.

Every production vector database, regardless of vendor, is built from the same four architectural layers stacked on top of each other.

┌─────────────────────────────────────────────┐
│              API / Query Layer               │
│   REST, gRPC, Python SDK, filter parsing     │
├─────────────────────────────────────────────┤
│              Index Layer                     │
│   HNSW, IVF, flat (brute force), PQ          │
├─────────────────────────────────────────────┤
│              Storage Layer                   │
│   Vector store, metadata store, WAL          │
├─────────────────────────────────────────────┤
│           Infrastructure Layer               │
│   Replication, sharding, compaction, GC      │
└─────────────────────────────────────────────┘

The API layer receives queries and writes, parses filters, and routes requests. The index layer implements the ANN algorithms that make similarity search fast. The storage layer persists vectors, metadata, and the index itself to disk. The infrastructure layer handles durability, distribution, and maintenance operations that keep the system healthy over time.

Understanding each layer independently is what allows you to reason about performance and correctness separately. A slow query is almost always an index layer problem. A missing result after a recent upsert is almost always a storage or compaction timing problem. A metadata filter that returns unexpected results is almost always an API layer parsing problem or a filtering strategy mismatch.

The Ingestion Pipeline: What Happens When You Write a Vector

A write operation in a vector database is more complex than it appears from the client side. A single upsert call triggers a sequence of operations across multiple layers.

from qdrant_client import QdrantClient, models

client = QdrantClient(host="localhost", port=6333)

client.upsert(
    collection_name="knowledge-base",
    points=[
        models.PointStruct(
            id=1,
            vector=[0.41, -0.22, 0.88, ...],   # 1536 floats
            payload={"text": "refund policy", "source": "docs/refund.md"}
        )
    ]
)

From the perspective of the database internals, this single call triggers the following sequence:

Client upsert request
        ↓
1. Validation
   Dimension check: vector must have exactly 1536 floats
   ID format check: must be valid integer or UUID
   Payload schema: no constraints in most vector DBs
        ↓
2. Write-Ahead Log (WAL) append
   Mutation serialized and written to WAL on disk
   WAL fsync: operation is now crash-safe
   Acknowledgment can now be returned to client
        ↓
3. In-memory segment write
   Vector bytes appended to mmap'd vector store
   Payload written to metadata key-value store
   Soft-delete bitmap updated (if this is an update)
        ↓
4. Background: index update
   New vector inserted into HNSW graph (incremental)
   Or: vector appended to unsealed IVF list
        ↓
5. Background: segment sealing and compaction
   When segment reaches size threshold → seal + build full HNSW
   Periodic compaction merges small segments
   Deleted vectors purged from storage

According to Zilliz's vector database internals guide, a vector database provides data storage, CRUD operations, metadata filtering, and horizontal scaling. The durability guarantee requires that at step 2, the WAL write and fsync completes before the client receives an acknowledgment. Everything after that is best-effort background work.

The WAL is what separates a vector database from a standalone vector index like FAISS. FAISS has no persistence mechanism. If the process crashes, all data is lost. A vector database writes every mutation to an append-only log before applying it, so a crashed process can replay the WAL on restart and recover to a consistent state.

The Storage Layer: How Vectors Are Persisted

Storing millions of high-dimensional float vectors efficiently requires a different approach than storing rows in a relational database.

Vector Storage Format

Vectors are stored as flat binary arrays of 32-bit floats (float32) or 16-bit floats (float16 for compression). A 1536-dimensional float32 vector occupies exactly 1536 × 4 = 6,144 bytes. Vectors for a collection are stored contiguously, which allows the ANN index to read sequential ranges of vectors efficiently using memory-mapped files (mmap).

import numpy as np
import struct

def serialize_vector(vec: list[float]) -> bytes:
    """Convert a Python float list to packed binary float32."""
    return struct.pack(f"{len(vec)}f", *vec)

def deserialize_vector(data: bytes) -> list[float]:
    """Convert packed binary float32 back to Python floats."""
    n = len(data) // 4
    return list(struct.unpack(f"{n}f", data))

# A 1536-dim vector
vec = [0.41, -0.22, 0.88, 0.03, -0.71, ...]   # 1536 elements

serialized = serialize_vector(vec)
print(f"Serialized size: {len(serialized)} bytes")
# Serialized size: 6144 bytes

recovered = deserialize_vector(serialized)
assert recovered == vec

The flat binary format is cache-friendly for the ANN index. When HNSW traverses the graph and computes the distance between two nodes, it reads both vectors from memory sequentially. Contiguous float storage maximizes CPU cache hit rates for those reads.

Metadata Storage

The payload (metadata) for each vector is stored separately from the vector bytes, typically in a column store or a key-value store indexed by vector ID. This separation allows the ANN index to operate entirely on float arrays without loading metadata, keeping the hot path for search as lean as possible.

# Logical separation between vector store and metadata store

# Vector store: ID → raw float bytes
vector_store = {
    1: serialize_vector([0.41, -0.22, 0.88, ...]),
    2: serialize_vector([-0.11, 0.73, -0.44, ...]),
}

# Metadata store: ID → structured payload
metadata_store = {
    1: {"text": "refund policy", "source": "docs/refund.md",   "category": "support"},
    2: {"text": "timeout errors", "source": "docs/errors.md",  "category": "engineering"},
}

# Index: float arrays only, stores IDs not payloads
# At query time: index returns [id=1, id=2], then metadata_store[1] fetched

According to WEKA's vector database architecture guide, the vector database persistently stores associated metadata with each data point alongside the vector in the storage layer. The structure is usually optimized for fast retrieval and low memory usage.

The Segment Model

Rather than maintaining one monolithic index over the entire collection, modern vector databases use a segment-based storage model. A segment is an immutable unit of storage containing a batch of vectors, their metadata, and an ANN index built specifically over that batch.

Collection: "knowledge-base" (5M vectors total)

Segment 1 (sealed, 1M vectors):
  vector_data.bin    → 6 GB of float32 arrays
  metadata.db        → 1M payload records
  hnsw_index.bin     → HNSW graph for these 1M vectors
  deleted.bitmap     → tracks which IDs are soft-deleted

Segment 2 (sealed, 1M vectors):
  [same structure]

Segment 3 (sealed, 1M vectors):
  [same structure]

Segment 4 (sealed, 1M vectors):
  [same structure]

Segment 5 (active, 1M vectors, growing):
  vector_data.bin    → vectors appended incrementally
  metadata.db        → payloads appended incrementally
  flat_index.bin     → brute-force search (no HNSW yet)
  wal.log            → uncommitted mutations

New writes always go into the active segment. The active segment uses brute-force (flat) search rather than HNSW because HNSW cannot efficiently insert vectors one at a time without rebuilding graph edges. When the active segment reaches a size threshold (typically 100,000 to 1,000,000 vectors), it is sealed, a full HNSW index is built over its contents, and a new active segment is opened.

At query time, all segments are searched in parallel and results are merged by score before the top-K are returned. The brute-force search on the small active segment is fast because it contains relatively few vectors. The HNSW search on sealed segments is fast because of the ANN index.

According to Chakra Dev's vector database introduction, Qdrant is designed for production with consistency and handles frequent updates well through its segment-based update model.

The Index Layer in the Architecture

The index layer sits between persisted vectors and the query API. Its job is to produce a small set of nearby candidate IDs without scanning the entire collection. The storage and segment layers decide which vectors are available; the index provides the search structure built over those vectors.

Production systems commonly use HNSW graphs, IVF clusters, flat indexes, or compressed variants such as IVF-PQ. From an architecture perspective, the important distinction is that each segment owns or references an index artifact that must be created, loaded, updated, and rebuilt during compaction.

For algorithms and tuning, read what vector indexing is, how HNSW works, how IVF works, and how product quantization compresses vectors.

Where Similarity Metrics Fit

The distance metric defines what the index treats as “near.” It is part of the collection configuration and must remain compatible with the embedding model and the index built over its vectors. Cosine similarity, Euclidean distance, and dot product are common choices, but their mathematics and selection rules are search concerns rather than storage architecture.

Architecturally, the metric affects index construction and every candidate comparison. Changing it generally requires rebuilding the index because the existing search structure was organized under the previous distance function. Read cosine similarity vs Euclidean distance for formulas, examples, and metric-selection guidance.

The Query Path Across the Architecture

A query crosses the layers in the opposite direction from a write. The API validates and routes the request, the index layer produces nearby candidate IDs, and the storage layer supplies the metadata or payload fields needed for the response. Segment- or shard-level results may be combined before the final response is returned.

API request → index candidates → storage lookup → assembled response

This pillar uses that flow only to show how the components connect. The vector query lifecycle guide owns the procedural sequence, distributed merging, serialization, latency attribution, and observability.

Metadata Filtering as a Cross-Layer Concern

Metadata filters restrict vector results using structured fields such as tenant, category, date, or access level. They cross architectural boundaries: the API parses the condition, the metadata store identifies or evaluates eligible records, and the search layer must respect that eligibility while finding nearby vectors.

Systems may evaluate filters before, during, or after candidate retrieval. That choice affects result completeness and search cost, but it does not change where the capability lives: filtering coordinates the API, index, and metadata storage layers. The vector query lifecycle guide explains filter strategy selection and execution in context.

Exact and Approximate Search in the Index Layer

Exact search compares a query with every eligible vector. Approximate search uses an index to avoid most comparisons and return a close estimate of the nearest neighbors. Architecturally, both are execution modes of the index layer: flat search can serve small or newly active segments, while ANN structures serve larger sealed segments.

The choice influences resource use and result recall, but tuning parameters and benchmarking belong outside this architecture overview. Read exact vs approximate nearest-neighbor search for the trade-off, measurement method, and configuration guidance.

Distributed Architecture: Scaling Beyond One Node

A single vector database node on modern hardware handles tens of millions of vectors comfortably. Beyond that threshold, or when write throughput or query concurrency exceeds single-node capacity, horizontal distribution is required.

Sharding

The collection is split into shards. Each shard holds a subset of the vectors and its own ANN index. Sharding is typically done by hash of the vector ID so that the assignment is deterministic and balanced across nodes.

Collection "knowledge-base" (100M vectors, 4 shards):

Shard 0 (Node A):  vectors where ID % 4 == 0  → 25M vectors + HNSW index
Shard 1 (Node B):  vectors where ID % 4 == 1  → 25M vectors + HNSW index
Shard 2 (Node C):  vectors where ID % 4 == 2  → 25M vectors + HNSW index
Shard 3 (Node D):  vectors where ID % 4 == 3  → 25M vectors + HNSW index

Query coordinator:
  1. Broadcast query to all 4 shards
  2. Collect top-K results from each shard
  3. Merge 4 × K results → global top-K
  4. Return final top-K to client

The merge step at the coordinator is where the global top-K is assembled. Each shard returns its local top-K sorted by score. The coordinator merges these K sorted lists into one global ranking and selects the top-K from the combined list.

Replication

Each shard is replicated across multiple nodes for fault tolerance. Reads can be served from any replica. Writes must be propagated to all replicas before the write is acknowledged (synchronous replication) or propagated asynchronously after acknowledgment (asynchronous replication). The choice determines whether the system is CP (consistent and partition-tolerant) or AP (available and partition-tolerant) in CAP theorem terms.

According to WEKA's architecture guide, vector databases scale with increasing data volume and user demands, with more support for parallel and distributed processing. The serverless architectures of vector databases also optimize cost at scale.

Milvus: Disaggregated Architecture

Milvus takes disaggregation further by separating storage, indexing, and query responsibilities into independent services:

Milvus disaggregated architecture:

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Query Nodes │    │  Data Nodes │    │ Index Nodes  │
│  (stateless) │    │ (ingestion) │    │(index build) │
└──────┬──────┘    └──────┬──────┘    └──────┬──────┘
       │                  │                   │
       └──────────────────┼───────────────────┘
                          │
              ┌───────────▼───────────┐
              │   Object Storage      │
              │  (S3 / MinIO / GCS)   │
              │  segments, indexes    │
              └───────────────────────┘

Query nodes are stateless. They load segment data from object storage on demand, serve queries, and can be scaled horizontally without affecting data durability. Index nodes build HNSW indexes over sealed segments asynchronously. Data nodes receive write traffic and manage the WAL. Object storage provides the durable, infinitely scalable backing store.

This disaggregated model allows each component to scale independently based on its bottleneck. If queries are slow, add query nodes. If ingestion is falling behind, add data nodes. If index build is lagging, add index nodes.

CRUD Operations: Updates and Deletes

Vector indexes are fundamentally append-friendly structures. Inserting a new vector into an HNSW graph requires adding a new node with edges to its nearest neighbors, which is relatively efficient. Deleting or updating a vector is not.

An HNSW graph node cannot be cleanly removed without potentially disconnecting the graph and degrading recall for other nodes. Most vector databases solve this with soft deletes: a vector is marked as deleted in a bitmap, and the ANN search skips deleted IDs when assembling results.

# Qdrant delete operation - soft delete under the hood
client.delete(
    collection_name="knowledge-base",
    points_selector=models.PointIdsList(points=[1, 2, 3]),
)

# What happens internally:
# 1. IDs [1, 2, 3] added to the deleted_ids bitmap for their segment
# 2. ANN search results containing these IDs are filtered out
# 3. The actual vector bytes remain in storage
# 4. During compaction: deleted vectors are physically removed
#    and the segment's HNSW index is rebuilt without them

Updates are implemented as a delete followed by an insert. The old vector is soft-deleted and a new vector (with potentially different float values and metadata) is inserted into the active segment. This means a "updated" record appears in two places temporarily: the soft-deleted old version in its original segment, and the new version in the active segment. Compaction eventually consolidates them.

Vector Indexing as an Architectural Asset

An index is a persisted artifact associated with a collection or segment. It has its own build process, memory footprint, storage footprint, and replacement lifecycle. New segments may use flat search until an ANN index is built; compaction can create a replacement segment and index before the old files are retired.

This architecture page treats the index as a managed system component. The dedicated vector indexing guide covers index types, parameter selection, quality measurement, and operational tuning.

Connecting the Cluster Articles

This pillar provides the architectural frame. Each cluster article goes deep on one component:

The similarity search mechanics article covers how the distance computation inside an ANN search actually runs, from the first graph traversal step to the final ranked list.

The cosine similarity vs Euclidean distance article works through the mathematics of both metrics with geometric intuition and code, and maps each to the embedding models where it is appropriate.

The exact vs approximate nearest neighbor article covers the precision-speed tradeoff formally, how to measure recall, and how ANN algorithms bound their approximation error.

The HNSW algorithm article covers the full graph construction and traversal algorithm with diagrams, parameter tuning, and the small-world network theory that explains why it works.

The IVF index article covers the k-means training phase, cluster assignment, the nprobe tradeoff, and when IVF beats HNSW. It cross-links with the HNSW article for the direct comparison.

The Product Quantization article covers the subspace decomposition algorithm, codebook training, asymmetric distance computation, and how PQ combines with IVF for billion-scale deployment.

The vector indexing article covers the indexing discipline as a whole: why indexes exist, what properties a good vector index must have, and how index quality is measured.

The vector query lifecycle article traces a single search request through every internal component from API parsing to response serialization, providing the observability frame for debugging production issues.

Summary

A vector database is not a vector index with a REST API bolted on. It is a complete data management system with at least four layers of engineering: the API and filter parsing layer, the ANN index layer (HNSW, IVF, flat), the storage layer (binary vector store, metadata key-value store, WAL, segments), and the infrastructure layer (replication, sharding, compaction).

Every write goes through the WAL before acknowledgment, ensuring crash recovery. Vectors are stored as contiguous float32 binary arrays. The segment model allows incremental ingestion without continuous index rebuilding. Deletes are soft to avoid corrupting the HNSW graph and are cleaned up by compaction.

On the read path, the API, index, metadata store, segments, and any distributed coordinators work together to assemble a response. The dedicated query and search articles explain the procedural mechanics inside that path.

The cluster articles in this series go deep on each component. The Vector Database Fundamentals series covers the foundational concepts that this series builds on.

Sources and Further Reading

1. Pinecone. What Is a Vector Database and How Does It Work? pinecone.io/learn/vector-database
2. Zilliz. What Is a Vector Database and How Does It Work? zilliz.com/learn/what-is-vector-database
3. Milvus. What Exactly Is a Vector Database? milvus.io/blog/what-is-a-vector-database.md
4. Qdrant. What Is a Vector Database? qdrant.tech/articles/what-is-a-vector-database
5. IBM. What Is a Vector Database? ibm.com/think/topics/vector-database
6. Databricks. What Is a Vector Database? databricks.com/blog/what-is-vector-database
7. WEKA. Vector Database: Everything You Need to Know. weka.io/learn/guide/ai-ml/vector-database
8. Yugabyte. What Is a Vector Database? Examples, Use Cases. yugabyte.com/blog/what-is-a-vector-database
9. Pure Storage. Vector Databases and Storage for Generative AI. blog.purestorage.com/purely-technical/vector-database-and-storage
10. arXiv. A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge. arxiv.org/abs/2310.11703
11. arXiv. Vector Search for the Future: From Memory-Resident to Cloud-Native Architectures. arxiv.org/abs/2601.01937
12. Weaviate. Vector Database Architecture. weaviate.io/developers/weaviate/concepts/vector-index