Day 27 Self Attention Explained From Scratch

Day 27 of a 30-day ML challenge asks you to understand self-attention from scratch. Self-attention is the core mechanism in transformer models like GPT and BERT; it lets each token in a sequence “look at” every other token to decide which ones matter most for its own representation. You don’t need a heavy math background to follow the basic idea—just a grasp of vectors, dot products, and softmax.

Formula

The attention weight from token i to token j is computed as:

attention(i,j) = softmax( (Q_i · K_j) / sqrt(d_k) )

where:

• Q_i is the query vector for token i (dimension d_k)
• K_j is the key vector for token j (dimension d_k)
• d_k is the dimension of the key/query vectors (often equal to d_model / num_heads)
• · denotes the dot product
• softmax is applied over all j for a fixed i, so the weights sum to 1

In practice, Q, K, and V are obtained by multiplying the input embeddings by learned weight matrices. The final output for token i is a weighted sum of the value vectors V_j, using the attention weights.

Worked Example

Let’s use the default values from the calculator: 4 tokens, embedding dimension 8. We’ll compute the attention weight from token 0 to token 1, assuming we already have Q_0 and K_1.

Suppose Q_0 = [0.5, -0.2, 0.1, 0.3, -0.4, 0.2, 0.0, 0.6] and K_1 = [0.8, 0.1, -0.3, 0.2, 0.5, -0.1, 0.4, 0.0]. The dot product is:

0.5*0.8 + (-0.2)*0.1 + 0.1*(-0.3) + 0.3*0.2 + (-0.4)*0.5 + 0.2*(-0.1) + 0.0*0.4 + 0.6*0.0 = 0.4 – 0.02 – 0.03 + 0.06 – 0.2 – 0.02 + 0 + 0 = 0.19

Assume d_k = 8, so we divide by sqrt(8) ≈ 2.828: 0.19 / 2.828 ≈ 0.0672.

Now we need the same scaled dot products for all other tokens (j=0,2,3). Suppose they are: 0.0672 (for j=1), 0.5, -0.3, 0.1. Apply softmax: compute exp of each: exp(0.0672)=1.0695, exp(0.5)=1.6487, exp(-0.3)=0.7408, exp(0.1)=1.1052. Sum = 4.5642. Then attention(0,1) = 1.0695 / 4.5642 ≈ 0.234, which rounds to 0.23. The calculator gives 0.25 because it uses a simplified example with different vectors; the process is what matters.

Why Self-Attention Was a Breakthrough

Before transformers, sequence models like RNNs and LSTMs processed tokens one by one, carrying a hidden state. That made it hard to capture long-range dependencies and forced sequential processing. Self-attention lets every token interact with every other token in a single step, so the model can directly relate words that are far apart. For example, in the sentence “The animal didn’t cross the street because it was too tired,” self-attention helps the model connect “it” to “animal” even though they are separated by several words.

The Steps in Detail

Here’s the full pipeline for a single attention head:

1. Create Q, K, V: Multiply the input embedding matrix X (shape: tokens × d_model) by three weight matrices W_Q, W_K, W_V (each d_model × d_k) to get Q, K, V.
2. Compute scores: For each pair (i, j), calculate the dot product Q_i · K_j. This gives a matrix of raw scores.
3. Scale: Divide each score by sqrt(d_k). This prevents the dot products from growing too large, which would push softmax into regions with tiny gradients.
4. Apply softmax: For each row i, apply softmax over all j, so each row sums to 1. This gives the attention weights.
5. Weighted sum: Multiply each V_j by the weight attention(i,j) and sum over j. The result is the output for token i.

Multi-Head Attention and Why It Matters

Instead of using one set of Q, K, V, transformers use multiple heads. Each head has its own weight matrices, so it can attend to different types of relationships. For instance, one head might focus on syntactic dependencies, another on coreference, and another on positional relationships. The outputs of all heads are concatenated and passed through a linear layer. This allows the model to capture multiple aspects of the input simultaneously, which is crucial for language understanding.

Common Misconceptions and Pitfalls

• Myth: Attention weights are interpretable. While they can sometimes align with human intuition, they are not reliable explanations. The weights depend on the learned matrices and can be counterintuitive.
• Mistake: Forgetting to scale. Without the 1/sqrt(d_k) factor, dot products can become very large, leading to softmax saturation and vanishing gradients.
• Mistake: Confusing d_model and d_k. In multi-head attention, d_k is usually d_model / num_heads, not d_model itself.
• Mistake: Thinking self-attention is order-agnostic. The attention mechanism itself doesn’t encode position, so positional encodings are added to the input embeddings to give the model a sense of order.

When to Use Self-Attention (and When Not)

Self-attention is the go-to for tasks where relationships between all elements matter, such as machine translation, text classification, and even image processing (vision transformers). However, it has a quadratic cost: for n tokens, the attention matrix is n×n, so memory and compute grow with n². For very long sequences (e.g., entire books), this becomes prohibitive. Alternatives like sparse attention, linear attention, or sliding window attention reduce the cost, but they often sacrifice some ability to capture long-range dependencies. For simple sequence tasks with short sequences, an RNN or a convolutional model might be more efficient.

Frequently Asked Questions

What is the difference between self-attention and regular attention?

Regular attention (as in seq2seq models) computes attention between the decoder’s query and the encoder’s keys/values. Self-attention computes attention within the same sequence: every token attends to every other token in that same sequence. This allows each token to incorporate context from its entire sentence or document.

Why do we divide by sqrt(d_k)?

This scaling prevents the dot products from becoming too large, which would push the softmax into a region where it produces near-one-hot distributions. That would make gradients very small and slow down training. The sqrt(d_k) scaling keeps the variance of the dot products roughly constant, which stabilizes training.

Can self-attention handle variable-length sequences?

Yes, the mechanism itself works with any length, because the weight matrices are fixed-size but the number of tokens can vary. However, the computational cost grows quadratically with length, and many implementations use padding or truncation to keep batches uniform.

How does positional encoding work?

Since self-attention is permutation-invariant (it doesn’t know the order of tokens), positional encodings are added to the input embeddings. These are vectors that encode the position of each token, often using sine and cosine functions of different frequencies. The model can then use these to attend based on relative or absolute positions.

What is the role of the value vector V?

The value vector contains the actual content that will be aggregated. The query and key determine the weights, but the output is a weighted sum of values. In a sense, Q asks “what am I looking for?”, K says “what do I contain?”, and V provides the information to pass along.

Is self-attention used only in NLP?

No, it’s used in computer vision (ViT), speech processing, and even reinforcement learning. Wherever you have a set of elements that interact, self-attention can model those interactions. It’s a general-purpose relational mechanism.

Related

• How to Change Galaxy S8 App Icon Size
• CSV Demo to Production AI Agent: Systems You Must Add
• AI Lethal Decisions: How to Tell if a System Decides Life or Death

in t | Number of tokens | - | 4 | 1
in d | Embedding dimension | - | 8 | 1
out attn | Attention weight (token 0 to token 1) | - | 0.25 | 2