05. Transformer LLM(Attention is All you need)

Transformer-based-llm Attention

It starts with the question

If attention works so well at highlighting important inputs, why do we need all these recurrent things to start with?

A model where everything attends to everything else

Overall Structure

Multi-head attention

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.

Motivation

Instead of performing a single attention function, we run multiple attention heads in parallel. This allows the model to capture different aspects of the relationships between tokens (e.g., syntax vs. semantics).

Formula Overview

Given:

• Input: Queries Q, Keys K, Values V
• Weight matrices: $W_h^Q,W_h^K,W_h^V$ for each head $h\in [1,H]$
• Output projection: $W^O$

For each head ( h ), we compute:

$$Q^{(h)}=Q{W}_h^Q$$ $$K^{(h)}=K{W}_h^K$$ $$V^{(h)}=V{W}_h^V$$

Q, K, V is

Perform scaled dot-product attention:

$$A^{(h)}=\text{softmax}\left(\frac{Q^{(h)}(K^{(h)}{)}^T}{\sqrt{d_k}}\right)V^{(h)}$$

Divide by $\sqrt{d_k}$ , so that we don’t have to hyperparam tune whenever we change dimension

Then concatenate all heads and project:

$$\text{MultiHead}(Q,K,V)=\text{Concat}(A^{(1)},\dots ,A^{(H)})W^O$$

Step-by-Step Breakdown

1. Linear projections of Q, K, and V using learned weights $W_h^Q$,$W_h^K$,$W_h^V$

2. Scaled dot-product attention is computed separately for each head

3. Concatenate all heads’ outputs

4. Final linear projection using $W^O$

Matrix Dimensions

Component	Shape
Q, K, V	(seq_len, d_model)
W^Q, W^K, W^V	(d_model, d_k or d_v)
Q^{(h)}, K^{(h)}	(seq_len, d_k)
Output per head	(seq_len, d_v)
Final output	(seq_len, H × d_v) → (seq_len, d_model) after W^O

Why Multiple Heads?

• Attend to different things (e.g., subject-verb agreement, word order, phrase-level context)

• Learn different subspace representations

• Aggregate richer and more diverse information

• Shown empirically to improve performance over single-head attention

Each attention head performs:

Q, K, V → Scaled Dot-Product Attention → $A^{(h)}$

Then:

$[A^{(1)}|A^{(2)}|...|A^{(H)}]\to W^O$ → Final Output

Layer Normalization

1. Compute mean and std of activation:
2. Subtract mean and divide by std

This way we can scale and shift, providing flexibility.

Why transformer architecture rather than RNN

1. Everything is feedforward: Parallel training, Full leverage of GPU
2. Every time step can attend to any others: No range limit of interaction, No more memory or vanishing gradient through time