One Hot Encoding vs Embeddings: How to Choose
The calculator for this page is switched off while it is checked. The formula is below, and the worked example in the article shows how to use it by hand.
One hot encoding and embeddings are two ways to turn categorical data into numbers for machine learning; choose one hot encoding when the number of categories is small and there is no natural ordering, and choose embeddings when the number of categories is large or the categories have a meaningful similarity structure. The right choice depends on the size of your dataset, the cardinality of the feature, and whether you need interpretability or predictive performance.
Formula
One hot encoding creates a binary vector of length C (number of categories), with a single 1 at the index of the category and 0 elsewhere. Embeddings map each category to a dense vector of dimension d (typically 2–50), learned during training. The number of parameters for an embedding layer is C × d, while one hot encoding has no learnable parameters but produces C-dimensional input vectors.
Worked Example
With 10 categories and 1000 samples, one hot encoding gives each sample a 10-dimensional vector, so the input matrix is 1000 × 10. If you choose an embedding dimension of 4, the embedding layer has 10 × 4 = 40 parameters. The ratio of embedding parameters to one hot vector size is 40 / 10 = 4.0, meaning the embedding is more compact per category but still introduces learnable weights.
Background: Why Categorical Data Needs Encoding
Machine learning models operate on numbers, not strings. A column with values like “red”, “green”, “blue” must be converted to numeric form. The simplest approach is label encoding (assigning 0, 1, 2), but that implies an artificial ordering: green becomes larger than red, which can mislead linear models. One hot encoding avoids that by creating a separate binary column per category. Embeddings go further by learning a continuous vector representation that captures similarity—categories that appear in similar contexts end up with similar vectors.
How One Hot Encoding Works
For a categorical feature with C unique values, one hot encoding produces a binary vector of length C. For example, if the categories are {cat, dog, bird}, then “cat” becomes [1,0,0], “dog” becomes [0,1,0], and “bird” becomes [0,0,1]. This is a sparse, high-dimensional representation. In practice, you can use a sparse matrix to save memory, but the dimensionality grows linearly with the number of categories.
How Embeddings Work
An embedding is a trainable lookup table: each category index points to a dense vector of fixed size. The vectors are initialized randomly and updated via backpropagation during training. The embedding dimension is a hyperparameter you choose—common values are 4, 8, 16, or 32. The key advantage is that the model learns which categories are similar for the task at hand. For example, in a recommendation system, embeddings for movies can capture genres even if you never explicitly provide that information.
How to Choose: Key Factors
- Cardinality (number of categories): If C is small (say, under 10 or 20), one hot encoding is simple and effective. If C is large (hundreds or thousands), embeddings are usually better because they keep the input dimension manageable and can generalize.
- Interpretability: One hot encoding is transparent—you can see exactly which category affects the prediction. Embeddings are opaque; you cannot easily explain why two categories have similar vectors.
- Dataset size: Embeddings require enough data to learn meaningful vectors. With very few samples, embeddings may overfit or fail to learn; one hot encoding is safer.
- Model type: Linear models (logistic regression, linear SVM) work well with one hot encoding. Neural networks can use either, but embeddings are designed for them.
- Memory and computation: One hot encoding with high cardinality creates huge sparse matrices. Embeddings are dense and compact, but add parameters to the model.
Common Problems and Pitfalls
A frequent mistake is using label encoding for ordinal data when the order is not meaningful—this can bias the model. Another issue is applying one hot encoding to high-cardinality features (e.g., user IDs) without dimensionality reduction, leading to the curse of dimensionality and overfitting. For embeddings, choosing too large a dimension can cause overfitting, while too small a dimension may lose information. Also, embeddings need to be trained; they do not work with non-neural models unless you pre-train them separately.
Costs and Trade-offs
One hot encoding has zero learnable parameters but increases the feature space, which can slow down training and require more memory. Embeddings reduce the input dimension but add parameters—each category gets a vector, so the total parameter count is C × d. For a categorical feature with 10,000 categories and dimension 32, that is 320,000 parameters, which is manageable but not trivial. The trade-off is often worth it for large-cardinality features because the model can share information across similar categories.
Alternatives to Both
If you have ordinal data (e.g., small, medium, large), label encoding with integer values preserves the order. Hashing tricks (feature hashing) can handle high-cardinality features by mapping categories to a fixed number of buckets, but collisions can hurt accuracy. Target encoding replaces each category with the mean target value for that category, which is useful for linear models but risks leakage if not done carefully. For natural language, pretrained word embeddings (like Word2Vec or GloVe) are often used instead of training embeddings from scratch.
Frequently Asked Questions
Can I use embeddings with non-neural models? Not directly. Embeddings are learned by neural networks. You could train an embedding layer in a neural network and then extract the vectors to feed into a random forest or logistic regression, but that is a two-step process.
What embedding dimension should I choose? A common heuristic is the fourth root of the number of categories (e.g., for 10,000 categories, dimension ≈ 10). In practice, values between 8 and 50 work well. Start small and increase if the model underfits.
Is one hot encoding always worse than embeddings? No. For small cardinality and small datasets, one hot encoding is often simpler, faster, and more interpretable. Embeddings shine when you have many categories and enough data to learn meaningful similarities.
How do I handle unseen categories during inference? For one hot encoding, an unseen category would produce an all-zero vector—this is a problem. For embeddings, you can use an “unknown” embedding vector that is learned or set to zero. Always reserve a special token for unseen categories during training.
Can I combine one hot encoding and embeddings? Yes. Some models use one hot encoding for low-cardinality features and embeddings for high-cardinality ones. This is common in recommendation systems where user IDs are embedded but categorical attributes like gender are one hot encoded.
Related
Formula used
in t | Number of categories | - | 10 | 2 in d | Training samples | - | 1000 | 10 in e | Embedding dimension (suggested) | - | 4 | 1 out ohe | One hot vectors per sample | - | t | 0 out emb | Embedding parameters | - | t * e | 0 out ratio | Embedding params / OHE params | - | (t * e) / t | 2
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.