Overview of the Transformer Architecture Data Flow
The Transformer architecture functions as a machine that predicts the next word in a sequence. It processes input text through a pipeline where the matrix size remains constant (e.g., 5 tokens $\times$ 768 dimensions) across stages to allow for easy stacking of blocks.
The three primary stages are:
- Embedding: Converts text strings into a numerical matrix. It maps tokens to vectors and adds positional information.
- Transformer Block (repeated $\times$N): Refines the numerical matrix. Each block consists of Self-Attention (to gather context from other words) and an MLP (to apply learned knowledge).
- Output: Converts the final vector of the last token into scores for the entire vocabulary, then into probabilities to select the next word.
Auto-Regressive Generation: This entire process is repeated for every single token predicted. Once a word is predicted, it is appended to the input and fed back into the model to predict the subsequent token.
"The cat sat on the" (Text)
↓ [Embedding]
[5 × 768] Matrix (Tokens × Dimensions)
↓ [Transformer Blocks 1-12]
[5 × 768] Refined Matrix
↓ [Output]
"floor" (Predicted Token)