Quantization is a model compression technique that converts high-precision numerical representations in a neural network—typically 32-bit or 16-bit floating-point numbers—into lower-precision formats such as 8-bit or 4-bit integers. This reduces model size, memory footprint, and computational cost, enabling deployment on resource-constrained hardware, at the expense of a small accuracy loss.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that freezes pre-trained model weights and injects small trainable low-rank matrices into each layer of the Transformer architecture. By learning only the difference between the original and adapted weights—approximated as the product of two much smaller matrices—LoRA reduces trainable parameters by up to 10,000x and GPU memory…

A foundation model is any model trained on broad data at scale—generally using self-supervision—that can be adapted to a wide range of downstream tasks. The term was introduced by Bommasani et al. (2021) at Stanford’s Center for Research on Foundation Models to describe models like BERT, GPT-3, and DALL-E that serve as a common basis…

Mixture of Experts (MoE) is a neural network architecture that replaces dense feed-forward layers with multiple specialized sub-networks called “experts” and uses a learned routing mechanism to selectively activate only a subset of those experts for each input. This enables models to be massive in total parameter count while keeping per-token computation lean.