Advanced · Generative AI
LoRA (Low-Rank Adaptation)
Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.
TL;DR. A parameter-efficient fine-tuning method injecting small trainable matrices into frozen pre-trained layers.
Technical Definition
A parameter-efficient fine-tuning method injecting small trainable matrices into frozen pre-trained layers.
How it works
LoRA freezes W and adds ΔW = BA where B and A are small matrices. With rank r=8 and d=4096, this adds 65K parameters instead of 16.7M. LoRA enables fine-tuning large models on consumer GPUs with zero additional inference latency.
Mathematical Notation
W' = W + BA where B ∈ ℝ^{d×r}, A ∈ ℝ^{r×d}Low-rank matrices B and A have far fewer parameters than the full weight matrix — a 256x reduction with r=8.
Related Concepts
- Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
- Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
- Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.