LoRA (Low-Rank Adaptation)
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that freezes pre-trained model weights and injects small trainable low-rank matrices into each layer of the Transformer architecture. By learning only the difference between the original and adapted weights—approximated as the product of two much smaller matrices—LoRA reduces trainable parameters by up to 10,000x and GPU memory…