Word2Vec: How It Learns Word Context

Word2Vec is a neural network technique that learns dense vector representations of words by predicting a word from its surrounding context (or vice versa). The core idea is that words appearing in similar contexts get similar vectors, so the model captures semantic and syntactic relationships directly from raw text.

Formula

The training objective for Word2Vec’s Skip-gram model is to maximize the average log probability of predicting each context word c given the center word w:

maximize (1/T) Σ_t=1..T Σ_{c∈context(t)} log p(c | w_t)

where T is the total number of words in the corpus, context(t) is the set of words within a window of size k around the center word w_t, and the probability is defined by the softmax function:

p(c | w) = exp(v_c · v_w) / Σ_u∈V exp(v_u · v_w)

Here, v_w is the input vector for the center word, v_c is the output vector for the context word, and V is the vocabulary size. Each word has two vectors: one as center and one as context. The final word embedding is typically the input vector.

Worked Example

Suppose we have a tiny corpus: “the cat sat on the mat”. With a window size of 1 (one word each side), the training pairs are:

• (cat, the), (cat, sat)
• (sat, cat), (sat, on)
• (on, sat), (on, the)
• (the, on), (the, mat)

For each pair, the model updates the vectors so that the probability of the context word given the center word increases. After many passes over the corpus, words that share contexts (like “cat” and “dog” if both appear with “pet”) end up with similar vectors. The result is that vector arithmetic works: v(“king”) – v(“man”) + v(“woman”) ≈ v(“queen”).

Background: Why Dense Vectors?

Before Word2Vec, text was often represented as bag-of-words or one-hot vectors. Those were sparse (thousands of dimensions) and lacked any notion of similarity: “car” and “automobile” were as different as “car” and “banana”. Word2Vec, introduced by Mikolov et al. at Google in 2013, solved this by learning low-dimensional (typically 100-300) dense vectors where distance reflects semantic relatedness. This made downstream tasks like clustering, classification, and search far more effective.

How Word2Vec Learns Context

Word2Vec comes in two architectures:

• Skip-gram: predicts context words from the center word. Works well with small corpora and rare words.
• CBOW (Continuous Bag-of-Words): predicts the center word from the average of context word vectors. Faster and better for frequent words on large corpora.

Training uses a shallow neural network with one hidden layer (no activation). The hidden layer weights are the embeddings. To make training tractable on large vocabularies, two optimizations are standard:

• Negative sampling: instead of computing softmax over all words, sample a few “negative” words and train a binary classifier that distinguishes real context words from noise.
• Hierarchical softmax: uses a Huffman tree over the vocabulary to reduce the output computation from O(V) to O(log V).

Choosing Parameters and Training Data

Key hyperparameters and their typical values:

• Vector size: 100-300. Larger captures more nuance but needs more data and memory.
• Window size: 5-10 for Skip-gram, 5 for CBOW. Smaller windows capture more syntactic similarity; larger windows capture more topical similarity.
• Minimum word count: discard words appearing fewer than 5-10 times to reduce noise.
• Subsampling: randomly drop frequent words (like “the”) with probability proportional to frequency, which speeds training and improves quality.
• Negative samples: 5-20 for small datasets, 2-5 for large ones.

Training data should be clean, lowercased (unless case matters), and tokenized. For English, common practice is to remove punctuation and numbers, but keep sentence boundaries so windows don’t cross them.

Common Problems and Misconceptions

A frequent misconception is that Word2Vec understands meaning. It does not; it only captures co-occurrence statistics. For example, “good” and “bad” often appear in similar contexts (“the movie was ___”), so they end up close in vector space, even though they are antonyms. This is why Word2Vec is good for similarity but not for sentiment.

Another issue is that embeddings are only as good as the corpus. A model trained on news articles will not capture slang from social media. Also, out-of-vocabulary words are simply ignored; there is no way to get a vector for a word not seen during training.

Finally, training on too little data or with too high a learning rate can produce degenerate embeddings where all words collapse to the same vector. Using a learning rate that decays over time and enough epochs (typically 3-5) helps.

Comparing Word2Vec to Other Embeddings

Word2Vec was a breakthrough, but it has limitations:

• GloVe: uses global word-word co-occurrence counts rather than local windows. Often produces similar quality but trains faster on large corpora.
• FastText: treats each word as a bag of character n-grams, so it can generate vectors for unseen words and better handles misspellings and rare words.
• Contextual embeddings (BERT, ELMo): produce different vectors for the same word depending on its sentence context (e.g., “bank” in “river bank” vs. “bank account”). Word2Vec gives one static vector per word, which is a major limitation for polysemy.

For many modern NLP tasks, contextual models outperform Word2Vec, but Word2Vec remains useful for its simplicity, speed, and when you need a fixed embedding for each word (e.g., in recommendation systems or when training data is limited).

Costs and Practical Considerations

Training Word2Vec on a large corpus (e.g., 100 billion words) requires significant CPU time (hours to days) and memory (the embedding matrix for a 1M vocabulary with 300 dimensions is about 1.2 GB). However, for most practical uses, you don’t train from scratch: pre-trained vectors are freely available for many languages and domains. Using these costs nothing and gives good baseline embeddings. If you need domain-specific vectors, training on a smaller corpus (e.g., 10 million words) can be done on a laptop in minutes.

Frequently Asked Questions

What is the difference between Skip-gram and CBOW?

Skip-gram predicts the surrounding words given the center word, while CBOW predicts the center word from the surrounding words. Skip-gram is slower but works better for rare words; CBOW is faster and better for frequent words. In practice, Skip-gram is often the default choice for smaller datasets.

How many dimensions should my word vectors have?

There is no universal answer, but 100-300 is common. More dimensions capture more distinctions but require more data and memory. If your corpus is small (under a few million words), 50-100 dimensions may be enough. You can also use pre-trained vectors with 300 dimensions and fine-tune them on your data.

Can Word2Vec handle multiple meanings of a word?

No. Word2Vec gives a single vector per word, so polysemy (e.g., “bank” as a financial institution vs. a river bank) is averaged into one representation. Contextual models like BERT solve this by producing a vector per occurrence, but they are much heavier to train and use.

Why do similar words end up close together?

Because the training objective forces words with similar contexts to have similar vectors. If two words frequently appear with the same neighboring words (e.g., “dog” and “cat” both appear with “pet”, “feed”, “walk”), the model adjusts their vectors to be similar to make predictions easier. This is the distributional hypothesis in action.

What is negative sampling exactly?

Instead of computing the softmax over the entire vocabulary (which is expensive), negative sampling turns the problem into a binary classification task: for each real context word, you also sample a few random words from the vocabulary (the “negative” samples) and train the model to output 1 for the real pair and 0 for the random pairs. This is much faster and works surprisingly well.

Can I train Word2Vec on my own data?

Yes. Libraries like Gensim (Python) provide a simple API: you pass a list of tokenized sentences and specify parameters like vector size, window, and min count. Training on a few million sentences takes minutes on a modern CPU. For best results, preprocess your text (lowercase, remove punctuation) and use a reasonable window size (5-10).

Related

• Screen Flashing Windows 10: Solved? Quick Fixes
• Delete IIS Log Files in Windows 10: Easy & Safe
• Pytest Plugins That Change How You Test

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2