RAM for 100M Embeddings: How Much Do You Need?

Storing 100 million embeddings typically requires between 20 GB and 80 GB of RAM, depending on the vector dimension and data type. For the most common setup—768-dimensional float32 vectors—you need about 307 GB, but using float16 or int8 quantization can cut that to roughly 154 GB or 77 GB respectively.

Formula

The RAM required is simply the product of the number of vectors, the vector dimension, and the bytes per value:

RAM (GB) = (N × D × B) / 1,073,741,824

• N = number of embeddings (e.g., 100,000,000)
• D = dimension of each embedding (e.g., 768 for BERT-base)
• B = bytes per value (4 for float32, 2 for float16, 1 for int8)
• 1,073,741,824 = number of bytes in a gigabyte (1024³)

Worked Example

Using the default values from the calculator: 100 million vectors, 768 dimensions, 4 bytes per value (float32).

1. Total values = 100,000,000 × 768 = 76,800,000,000
2. Total bytes = 76,800,000,000 × 4 = 307,200,000,000
3. Convert to GB: 307,200,000,000 / 1,073,741,824 ≈ 286.1 GB

The calculator shows 286.10 GB. Note that this is the raw data size; in practice, you’ll need extra memory for indexes, overhead, and the application itself.

Why Dimension and Data Type Matter

Embeddings come in many shapes. A small model like GloVe-50 uses 50 dimensions, while large language models like GPT-3 use 12,288. The data type also changes the footprint dramatically:

• float32 (4 bytes): standard precision, most compatible
• float16 (2 bytes): half the memory, slight precision loss
• int8 (1 byte): quarter the memory, more precision loss but often acceptable for similarity search

For example, 100M 384-dim embeddings (common for sentence transformers) in float32 need about 143 GB, but in int8 only 36 GB.

Real-World Scenarios

Here are typical use cases and their memory footprints for 100M vectors:

• Recommendation systems: user/item embeddings often 64–256 dims → 24–95 GB in float32
• Semantic search: sentence embeddings 384–1024 dims → 143–381 GB in float32
• Image retrieval: 512–2048 dims → 190–762 GB in float32

These numbers assume you load everything into RAM. If you use a disk-based index or a vector database, the RAM requirement can be much lower.

How to Reduce Memory Usage

If the raw calculation exceeds your available RAM, consider these options:

1. Use a smaller data type: switch from float32 to float16 or int8. This is the easiest win.
2. Reduce dimensionality: apply PCA or use a model with smaller output (e.g., 384 instead of 768).
3. Use a vector database: tools like FAISS, Annoy, or Milvus can store vectors on disk and only load part into RAM.
4. Shard across machines: distribute the vectors across multiple servers.
5. Use product quantization: compress vectors further (e.g., to a few bytes per vector) at the cost of some accuracy.

Common Mistakes and Misconceptions

One frequent error is forgetting that the formula gives the raw data size, not the total memory the application will use. Indexes, metadata, and the program itself add overhead—often 10–30%.

Another misconception is that you need to store all embeddings in RAM. For many applications, especially search, you can use approximate nearest neighbor (ANN) indexes that keep only a fraction of vectors in memory.

Also, be careful with units: some libraries report sizes in MiB (1024² bytes) while others use MB (1000²). The calculator uses GiB (1024³), which is the standard for RAM.

Frequently Asked Questions

How much RAM do 100 million 768-dim float32 embeddings need?

Exactly 286.1 GB for the raw data. Add overhead for the index and application, so plan for at least 300 GB.

Can I store 100 million embeddings in 64 GB of RAM?

Yes, if you use int8 quantization and keep dimensions below 512. For example, 100M × 384 × 1 byte = 38.4 GB, leaving room for overhead.

What is the difference between RAM and storage for embeddings?

RAM is fast, volatile memory used during computation; storage (SSD/HDD) is persistent but slower. You can store embeddings on disk and load them into RAM as needed, but search will be slower if you don’t keep them in memory.

Do I need to load all embeddings into RAM at once?

No. Many vector databases and libraries support memory mapping or disk-based indexes, allowing you to work with datasets larger than RAM.

How does the dimension of an embedding affect RAM?

RAM scales linearly with dimension. Doubling the dimension doubles the memory required for the same number of vectors.

What is the best data type for embeddings?

float32 is the safest for accuracy, but float16 and int8 are common for production systems where memory is limited. Test your specific use case to see if quality degrades.

Related

• Gemini 2 Preferences Settings: How to Adjust
• Scaling Test Automation with Hybrid Testing
• How to Save Documents Directly to Cloud

in d | Vector dimension | dims | 768 | 1
in n | Number of vectors | millions | 100 | 1
in b | Bytes per value | bytes | 4 | 1
out ram_gb | RAM needed | GB | (d * n * 1000000 * b) / (1024^3) | 2