Advanced · Neural Networks

Low-Rank Adaptation (LoRA)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A parameter-efficient fine-tuning technique that reduces computational cost by only training a small number of new parameters.

Technical Definition

A parameter-efficient fine-tuning technique that reduces computational cost by only training a small number of new parameters.

How it works

LoRA is a method for adapting large pre-trained models to new tasks without retraining the entire model. It injects small, trainable low-rank matrices into the transformer layers and only optimizes these new parameters. This drastically reduces the number of trainable parameters and VRAM requirements, making fine-tuning more accessible and efficient for specific applications.

Related Concepts

• Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
• Transfer Learning — Leveraging knowledge from a model trained on one task to improve performance on a different but related task.